Nov 03 00:02:23.470516 ip-10-0-143-162 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 00:02:23.470527 ip-10-0-143-162 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 00:02:23.470536 ip-10-0-143-162 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 00:02:23.470857 ip-10-0-143-162 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 00:02:33.556784 ip-10-0-143-162 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 00:02:33.556804 ip-10-0-143-162 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot da0b9dc204a342449c00eae3076f2fe1 -- Nov 03 00:04:26.718903 ip-10-0-143-162 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:04:27.051267 ip-10-0-143-162 kubenswrapper[2573]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:27.051267 ip-10-0-143-162 kubenswrapper[2573]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:04:27.051267 ip-10-0-143-162 kubenswrapper[2573]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:27.051267 ip-10-0-143-162 kubenswrapper[2573]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:04:27.056426 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056350 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.056426 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056425 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.056426 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056430 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056434 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056437 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056440 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056443 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056446 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056449 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056452 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056455 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056458 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056460 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056463 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056465 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056467 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056470 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056473 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056476 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056478 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056480 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056483 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.056558 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056485 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056495 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056498 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056500 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056503 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056506 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056508 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056511 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056513 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056516 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056519 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056522 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056524 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056527 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056530 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056533 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056535 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056538 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056540 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056543 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.057892 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056545 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056547 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056550 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056552 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056555 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056557 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056561 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056566 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056571 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056576 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056580 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056584 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056588 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056592 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056596 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056599 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056601 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056604 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056606 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.058342 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056609 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056611 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056614 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056617 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056620 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056622 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056625 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056639 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056642 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056645 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056647 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056650 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056652 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056655 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056657 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056660 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056662 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056665 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056667 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056670 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.058813 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056672 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056675 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056679 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056681 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056684 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056686 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056689 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056691 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056693 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056696 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056698 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056701 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056703 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056706 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056709 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056711 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056714 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056716 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056719 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.059263 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056723 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056728 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056731 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056734 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056737 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.056739 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057543 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057551 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057554 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057557 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057559 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057562 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057564 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057568 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057570 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057585 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057589 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057591 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057594 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057596 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.059718 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057599 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057602 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057604 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057607 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057609 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057612 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057614 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057618 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057620 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057623 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057625 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057640 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057644 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057647 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057649 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057651 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057655 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057659 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057662 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.060175 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057665 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057668 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057670 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057673 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057675 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057677 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057680 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057682 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057684 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057687 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057690 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057693 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057696 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057698 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057701 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057703 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057706 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057708 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057711 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057713 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.060606 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057716 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057718 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057720 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057723 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057725 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057728 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057730 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057733 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057735 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057738 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057740 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057743 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057745 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057748 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057750 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057752 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057754 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057757 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057759 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.061138 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057761 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057764 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057766 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057768 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057771 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057775 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057778 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057780 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057783 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057785 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057788 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057790 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057798 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057803 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057805 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057808 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057810 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057813 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057816 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057818 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.061591 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057821 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057824 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057826 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057829 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057831 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057834 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057836 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057839 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057842 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057844 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057846 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057849 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057853 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.057856 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058330 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058335 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058339 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058342 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058345 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058347 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.062193 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058350 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058353 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058355 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058358 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058361 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058363 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058365 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058368 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058370 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058373 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058376 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058378 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058381 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058384 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058386 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058389 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058391 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058394 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058396 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.062714 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058398 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058401 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058403 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058406 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058408 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058410 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058413 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058417 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058421 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058424 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058427 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058430 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058433 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058435 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058438 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058440 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058443 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058445 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058447 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.063202 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058450 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058452 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058455 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058457 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058461 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058464 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058466 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058469 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058471 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058473 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058476 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058478 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058480 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058483 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058485 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058487 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058490 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058492 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058495 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058497 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.063653 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058501 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058504 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058507 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058509 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058513 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058515 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058518 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058520 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058523 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058525 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058528 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058530 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058532 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058535 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058537 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058540 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058544 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058546 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058550 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.064108 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058554 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058556 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058559 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058561 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058564 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058567 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058569 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058571 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058574 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058576 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058578 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058581 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058583 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058586 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058589 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058592 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058594 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058597 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058600 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058603 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.064603 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058605 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058608 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.058610 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058720 2573 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058731 2573 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058744 2573 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058748 2573 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058752 2573 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058755 2573 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058761 2573 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058765 2573 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058768 2573 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058771 2573 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058774 2573 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058777 2573 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058780 2573 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058783 2573 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058786 2573 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058789 2573 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058792 2573 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058795 2573 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058801 2573 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058803 2573 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058806 2573 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:04:27.065091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058809 2573 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058813 2573 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058817 2573 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058820 2573 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058823 2573 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058827 2573 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058830 2573 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058833 2573 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058836 2573 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058839 2573 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058842 2573 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058846 2573 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058849 2573 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058852 2573 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058854 2573 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058857 2573 flags.go:64] FLAG: --enable-server="true" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058860 2573 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058865 2573 flags.go:64] FLAG: --event-burst="100" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058869 2573 flags.go:64] FLAG: --event-qps="50" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058871 2573 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058874 2573 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058877 2573 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058881 2573 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058884 2573 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058886 2573 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:04:27.065640 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058889 2573 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058892 2573 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058895 2573 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058898 2573 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058900 2573 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058903 2573 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058906 2573 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058909 2573 flags.go:64] FLAG: --feature-gates="" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058913 2573 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058916 2573 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058919 2573 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058922 2573 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058924 2573 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058927 2573 flags.go:64] FLAG: --help="false" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058930 2573 flags.go:64] FLAG: --hostname-override="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058934 2573 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058936 2573 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058939 2573 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058942 2573 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058946 2573 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058948 2573 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058951 2573 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058954 2573 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:04:27.066242 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058956 2573 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058959 2573 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058962 2573 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058965 2573 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058968 2573 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058970 2573 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058973 2573 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058976 2573 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058978 2573 flags.go:64] FLAG: --lock-file="" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058981 2573 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058984 2573 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058986 2573 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058991 2573 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058994 2573 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058997 2573 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.058999 2573 flags.go:64] FLAG: --logging-format="text" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059002 2573 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059006 2573 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059008 2573 flags.go:64] FLAG: --manifest-url="" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059011 2573 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059016 2573 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059018 2573 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059023 2573 flags.go:64] FLAG: --max-pods="110" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059025 2573 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059028 2573 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:04:27.066780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059031 2573 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059034 2573 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059037 2573 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059040 2573 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059043 2573 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059049 2573 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059052 2573 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059055 2573 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059058 2573 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059061 2573 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059064 2573 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059067 2573 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059069 2573 flags.go:64] FLAG: --port="10250" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059072 2573 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059075 2573 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0865646be73cbdacd" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059078 2573 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059081 2573 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059084 2573 flags.go:64] FLAG: --register-node="true" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059086 2573 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059090 2573 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059093 2573 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059095 2573 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059098 2573 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059102 2573 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059105 2573 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059108 2573 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:04:27.067343 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059111 2573 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059113 2573 flags.go:64] FLAG: --runonce="false" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059116 2573 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059119 2573 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059122 2573 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059124 2573 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059127 2573 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059131 2573 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059134 2573 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059136 2573 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059139 2573 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059142 2573 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059144 2573 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059147 2573 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059150 2573 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059152 2573 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059155 2573 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059160 2573 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059162 2573 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059165 2573 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059168 2573 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059171 2573 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059174 2573 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059177 2573 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059180 2573 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:04:27.067975 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059182 2573 flags.go:64] FLAG: --v="2" Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059186 2573 flags.go:64] FLAG: --version="false" Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059190 2573 flags.go:64] FLAG: --vmodule="" Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059194 2573 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059197 2573 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059327 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059331 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059334 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059337 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059339 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059342 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059344 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059347 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059349 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059352 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059359 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059362 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059365 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059367 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059369 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059372 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.068532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059375 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059378 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059380 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059383 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059385 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059387 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059390 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059392 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059395 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059397 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059400 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059405 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059408 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059411 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059414 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059418 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059421 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059425 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059427 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.069021 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059430 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059432 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059435 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059437 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059439 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059442 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059444 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059447 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059454 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059457 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059459 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059462 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059464 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059468 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059471 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059473 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059476 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059478 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059480 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059483 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.069458 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059485 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059488 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059490 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059493 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059496 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059499 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059502 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059504 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059506 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059509 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059511 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059514 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059516 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059518 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059521 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059523 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059526 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059528 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059531 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.069937 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059533 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059535 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059538 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059541 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059543 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059546 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059549 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059551 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059554 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059556 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059559 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059562 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059564 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059567 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059569 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059571 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059574 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059578 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059580 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059583 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.070360 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059585 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059588 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059590 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059592 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059595 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059597 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059600 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059602 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059605 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059607 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059609 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059612 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.059663 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059703 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059708 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059713 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.070826 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059717 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059720 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059726 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059728 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059731 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059734 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059736 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059739 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059741 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059744 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059746 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059749 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059751 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059753 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059757 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059760 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059762 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059765 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059767 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059770 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.071199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059772 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059774 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059777 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059779 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059782 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059784 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059787 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059789 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059791 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059794 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059797 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059799 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059802 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059804 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059811 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059813 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059816 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059818 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059821 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.071743 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059824 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059826 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059828 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059831 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059833 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059835 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059838 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059841 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059844 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059846 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059849 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059851 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059854 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059856 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059859 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059861 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059863 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059866 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059868 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.072212 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059871 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059873 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059876 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059878 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059880 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059883 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059885 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059888 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059892 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059894 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059896 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059899 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059901 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059903 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059906 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059908 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059911 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059913 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059916 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059918 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.072686 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059921 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059925 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059927 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059930 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059932 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059934 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059937 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059939 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059942 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059944 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059947 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059949 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059951 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059969 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059973 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059976 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059978 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059981 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059983 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059986 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.073172 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059990 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059994 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.059997 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.060000 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.060004 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.067045 2573 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.067059 2573 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.068460 2573 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.068467 2573 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068524 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068530 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068533 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068537 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068539 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068542 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068545 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068548 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068550 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068553 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.073621 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068555 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068558 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068561 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068563 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068566 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068568 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068570 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068573 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068576 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068578 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068581 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068584 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068586 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068589 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068591 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068594 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068596 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068599 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068601 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068604 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.074094 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068607 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068609 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068612 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068614 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068617 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068620 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068622 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068624 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068639 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068642 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068645 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068647 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068649 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068652 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068654 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068657 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068659 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068661 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068664 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.074545 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068667 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068669 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068671 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068674 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068676 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068679 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068681 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068684 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068687 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068689 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068692 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068694 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068697 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068700 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068702 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068705 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068707 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068710 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068712 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.074987 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068714 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068717 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068719 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068722 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068724 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068728 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068732 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068735 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068738 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068740 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068743 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068745 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068747 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068751 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068754 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068757 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068760 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068763 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.075413 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068765 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068768 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068770 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068773 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068775 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068778 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068780 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068784 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068786 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068789 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068791 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068794 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068796 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068798 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068801 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068803 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068805 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068808 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068810 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.075872 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068812 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.068839 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068969 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068974 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068977 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068980 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068982 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.068985 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069004 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069008 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069011 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069014 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069017 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069019 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069021 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069024 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.076643 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069027 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069030 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069033 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069035 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069038 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069041 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069043 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069045 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069048 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069050 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069053 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069055 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069058 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069060 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069062 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069065 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069067 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069070 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069072 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069074 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.077260 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069077 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069079 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069081 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069084 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069086 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069088 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069091 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069093 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069095 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069098 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069100 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069103 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069105 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069107 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069110 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069113 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069115 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069118 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069120 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.077740 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069123 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069125 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069127 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069130 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069133 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069137 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069139 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069142 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069145 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069147 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069150 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069152 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069155 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069157 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069159 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069162 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069164 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069166 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069169 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069171 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.078199 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069173 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069176 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069178 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069181 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069183 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069185 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069188 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069190 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069193 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069195 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069198 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069201 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069203 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069205 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069208 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069210 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069212 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069215 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069217 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069220 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.078701 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069222 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069224 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069227 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069229 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069233 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069237 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069240 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069242 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069245 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069247 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069250 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069254 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069256 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.069278 2573 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069306 2573 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:04:27.079158 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069310 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069313 2573 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069316 2573 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069318 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069320 2573 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069323 2573 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069326 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069328 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069331 2573 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069333 2573 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069335 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069338 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069340 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069342 2573 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069345 2573 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069347 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069349 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069352 2573 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069355 2573 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069357 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:04:27.079547 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069359 2573 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069362 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069364 2573 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069367 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069369 2573 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069371 2573 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069374 2573 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069376 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069378 2573 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069382 2573 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069384 2573 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069386 2573 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069389 2573 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069391 2573 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069394 2573 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069396 2573 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069398 2573 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069401 2573 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069404 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069407 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:04:27.080115 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069410 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069412 2573 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069415 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069417 2573 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069419 2573 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069421 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069424 2573 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069426 2573 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069429 2573 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069431 2573 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069433 2573 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069437 2573 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069440 2573 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069443 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069445 2573 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069448 2573 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069450 2573 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069452 2573 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069455 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:04:27.080588 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069457 2573 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069460 2573 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069464 2573 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069466 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069469 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069471 2573 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069474 2573 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069476 2573 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069479 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069481 2573 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069483 2573 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069489 2573 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069492 2573 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069494 2573 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069497 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069499 2573 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069502 2573 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069504 2573 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069507 2573 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069509 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:04:27.081039 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069512 2573 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069514 2573 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069516 2573 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069519 2573 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069521 2573 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069523 2573 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069526 2573 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069528 2573 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069530 2573 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069533 2573 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069535 2573 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069538 2573 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069541 2573 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069545 2573 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069549 2573 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069551 2573 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069554 2573 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069556 2573 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069559 2573 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:04:27.081532 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069562 2573 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069564 2573 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069566 2573 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069569 2573 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069572 2573 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069576 2573 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:27.069578 2573 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.069624 2573 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.071661 2573 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.072487 2573 server.go:1014] "Starting client certificate rotation" Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.072578 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.073138 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.074050 2573 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:04:27.081982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.077923 2573 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:04:27.090590 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.090575 2573 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:04:27.093281 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.093262 2573 log.go:25] "Validated CRI v1 image API" Nov 03 00:04:27.094768 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.094746 2573 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:04:27.097046 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.097026 2573 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ff44378-c6a0-433d-97b7-c71042d66846:/dev/nvme0n1p4 ea3afd72-ac51-4520-8743-ce188ec0d9f0:/dev/nvme0n1p3] Nov 03 00:04:27.097092 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.097046 2573 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 00:04:27.101859 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.101745 2573 manager.go:217] Machine: {Timestamp:2026-11-03 00:04:27.100496674 +0000 UTC m=+0.284154643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200635 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269f13b1ded54a6f3092b0b2fd822c SystemUUID:ec269f13-b1de-d54a-6f30-92b0b2fd822c BootID:da0b9dc2-04a3-4244-9c00-eae3076f2fe1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:03:62:05:11:97 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:03:62:05:11:97 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:63:95:4f:b9:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:04:27.101859 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.101858 2573 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:04:27.101964 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.101928 2573 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:04:27.104360 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.104341 2573 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:04:27.104480 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.104362 2573 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-162.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:04:27.104525 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.104482 2573 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:04:27.104525 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.104489 2573 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:04:27.104525 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.104521 2573 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:04:27.104594 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.104538 2573 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:04:27.105232 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.105222 2573 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:04:27.109060 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109048 2573 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:04:27.109171 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109162 2573 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:04:27.109781 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109771 2573 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:04:27.109814 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109787 2573 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:04:27.109814 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109803 2573 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:04:27.109814 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109813 2573 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:04:27.109898 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.109821 2573 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:04:27.110190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.110171 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:04:27.112311 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.112292 2573 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:04:27.116938 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.116925 2573 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:04:27.117693 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117681 2573 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:04:27.117724 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117711 2573 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:04:27.117870 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117859 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:04:27.117900 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117880 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:04:27.117900 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117889 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:04:27.117900 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117897 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:04:27.117973 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117913 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:04:27.117973 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.117922 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:04:27.118346 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.118337 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:04:27.118373 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.118346 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:04:27.118373 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.118354 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:04:27.118373 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.118359 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:04:27.118373 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.118372 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:04:27.118487 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.118381 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:04:27.119168 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.119160 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:04:27.119168 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.119169 2573 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:04:27.122749 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.122736 2573 server.go:1263] "Started kubelet" Nov 03 00:04:27.122819 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.122796 2573 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:04:27.122875 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.122826 2573 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:04:27.122926 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.122904 2573 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:04:27.125509 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.125486 2573 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:04:27.125650 ip-10-0-143-162 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:04:27.127457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.127440 2573 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:04:27.130282 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.130255 2573 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 00:04:27.134027 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.134005 2573 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:04:27.134381 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.134364 2573 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:04:27.134686 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.134665 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:27.135045 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135027 2573 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:04:27.135045 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135031 2573 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:04:27.135145 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135066 2573 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:04:27.135145 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135108 2573 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:04:27.135145 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135119 2573 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:04:27.135241 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135186 2573 factory.go:55] Registering systemd factory Nov 03 00:04:27.135241 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135197 2573 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:04:27.135408 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135394 2573 factory.go:153] Registering CRI-O factory Nov 03 00:04:27.135408 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135408 2573 factory.go:223] Registration of the crio container factory successfully Nov 03 00:04:27.135487 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135454 2573 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:04:27.135487 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135468 2573 factory.go:103] Registering Raw factory Nov 03 00:04:27.135487 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.135477 2573 manager.go:1199] Started watching for new ooms in manager Nov 03 00:04:27.136281 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.136265 2573 manager.go:319] Starting recovery of all containers Nov 03 00:04:27.147515 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.147501 2573 manager.go:324] Recovery completed Nov 03 00:04:27.151599 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.151587 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.153177 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153159 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.153234 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153198 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.153234 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153208 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.153693 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153672 2573 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:04:27.153693 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153692 2573 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:04:27.153779 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153707 2573 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:04:27.153779 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.153678 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.156027 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.156002 2573 policy_none.go:50] "Start" Nov 03 00:04:27.156103 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.156049 2573 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:04:27.156103 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.156067 2573 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:04:27.156354 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.156334 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.156453 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.156431 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.156542 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.156470 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.158290 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.158276 2573 policy_none.go:44] "Start" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.197154 2573 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199354 2573 manager.go:341] "Starting Device Plugin manager" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.199382 2573 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199398 2573 server.go:87] "Starting device plugin registration server" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199610 2573 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199620 2573 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199753 2573 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199842 2573 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.199851 2573 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.200870 2573 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.200968 2573 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.203624 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.204835 2573 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.204864 2573 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.204885 2573 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:04:27.213499 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.204924 2573 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 00:04:27.268505 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.268373 2573 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-143-162.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 00:04:27.300392 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.300374 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.301089 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.301072 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.301177 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.301100 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.301177 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.301110 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.301177 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.301130 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.305394 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.305361 2573 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal"] Nov 03 00:04:27.305438 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.305407 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.306085 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.306063 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.306085 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.306084 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.306163 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.306093 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.308090 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308079 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.308235 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308221 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.308270 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308248 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.308763 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308747 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.308856 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308775 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.308856 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308766 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.308856 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308789 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.308856 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308798 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.308856 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.308807 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.310868 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.310852 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.310950 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.310882 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.311531 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.311515 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.311704 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.311545 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.311704 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.311558 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.316669 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.316651 2573 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-143-162.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 00:04:27.317356 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.316578 2573 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-143-162.us-west-1.compute.internal.18e46025d026854e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-162.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-143-162.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 00:04:27.122541902 +0000 UTC m=+0.306199870,LastTimestamp:2026-11-03 00:04:27.122541902 +0000 UTC m=+0.306199870,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-162.us-west-1.compute.internal,}" Nov 03 00:04:27.333941 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.333918 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.335956 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.335941 2573 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.336005 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.335953 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.336005 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.335973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.336005 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.335989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b8cde6519a41e629e9c7fa53310b7cb-config\") pod \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"6b8cde6519a41e629e9c7fa53310b7cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.345868 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.345849 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:04:27.345931 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.345896 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z87mm" Nov 03 00:04:27.365329 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.365299 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" Nov 03 00:04:27.365329 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.365312 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Nov 03 00:04:27.365329 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.365302 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Nov 03 00:04:27.365329 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.365307 2573 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-143-162.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Nov 03 00:04:27.426811 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.426791 2573 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-143-162.us-west-1.compute.internal\" not found" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.436956 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.436938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.437032 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.436960 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.437032 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.436975 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b8cde6519a41e629e9c7fa53310b7cb-config\") pod \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"6b8cde6519a41e629e9c7fa53310b7cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.437032 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.437022 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b8cde6519a41e629e9c7fa53310b7cb-config\") pod \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"6b8cde6519a41e629e9c7fa53310b7cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.437120 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.437035 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.437120 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.437045 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.619476 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.619430 2573 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-143-162.us-west-1.compute.internal\" not found" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.627537 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.627522 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:27.628297 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.628283 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:27.628363 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.628310 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:27.628363 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.628320 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:27.628363 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.628342 2573 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.636533 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.636520 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.639492 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.639479 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.685693 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:27.685674 2573 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:27.685693 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.685692 2573 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-143-162.us-west-1.compute.internal\": node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:27.797690 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.797665 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:27.898198 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.898145 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:27.998677 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:27.998651 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.072747 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.072727 2573 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 00:04:28.073098 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.072847 2573 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://a94de1dd7bb8b42158c74d34dda92279-9029fc03a8717821.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-143-162.us-west-1.compute.internal.18e46025d1fa5cd5\": read tcp 10.0.143.162:54118->3.12.35.53:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-143-162.us-west-1.compute.internal.18e46025d1fa5cd5 default 4801 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-143-162.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-143-162.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-143-162.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 00:04:27 +0000 UTC,LastTimestamp:2026-11-03 00:04:27.306088462 +0000 UTC m=+0.489746430,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-143-162.us-west-1.compute.internal,}" Nov 03 00:04:28.099014 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.098994 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.199754 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.199706 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.300381 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.300361 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.335374 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.335354 2573 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:04:28.336071 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.336056 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:04:28.336129 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.336084 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:04:28.336129 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.336094 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:04:28.350581 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.350557 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:27 +0000 UTC" deadline="2028-05-20 10:19:25.238625993 +0000 UTC" Nov 03 00:04:28.350581 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.350578 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13546h14m56.888049936s" Nov 03 00:04:28.400703 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.400679 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.424173 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.424151 2573 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:28.500986 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.500945 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.521956 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.521937 2573 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:28.601158 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.601138 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.626782 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.626766 2573 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:28.702209 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:28.702193 2573 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:04:28.791664 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.791620 2573 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:04:28.835060 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.835036 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:28.896069 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.896038 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:04:28.897210 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.897194 2573 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:04:28.898043 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:28.898014 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8cde6519a41e629e9c7fa53310b7cb.slice/crio-e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55 WatchSource:0}: Error finding container e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55: Status 404 returned error can't find the container with id e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55 Nov 03 00:04:28.898742 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:28.898724 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a0942cf01c78617561368216297af20.slice/crio-a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4 WatchSource:0}: Error finding container a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4: Status 404 returned error can't find the container with id a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4 Nov 03 00:04:28.903524 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.903509 2573 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:04:28.953445 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:28.953426 2573 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:04:29.110423 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.110373 2573 apiserver.go:52] "Watching apiserver" Nov 03 00:04:29.134840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.134814 2573 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 00:04:29.208934 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.208894 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"6b8cde6519a41e629e9c7fa53310b7cb","Type":"ContainerStarted","Data":"e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55"} Nov 03 00:04:29.209981 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.209961 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"2a0942cf01c78617561368216297af20","Type":"ContainerStarted","Data":"a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4"} Nov 03 00:04:29.212441 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.212427 2573 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:04:29.212767 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.212752 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-cw8kk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn","openshift-multus/multus-additional-cni-plugins-5mqcz","openshift-network-diagnostics/network-check-target-gbt9l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal","openshift-multus/multus-wbjwv","openshift-multus/network-metrics-daemon-lcd5z","openshift-ovn-kubernetes/ovnkube-node-8lq8f","kube-system/konnectivity-agent-z8d7r","openshift-image-registry/node-ca-dgbsh","kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-lfw72","openshift-dns/node-resolver-4vplk","openshift-network-operator/iptables-alerter-t4jtn"] Nov 03 00:04:29.216408 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.216391 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.216488 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.216471 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:29.218459 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.218444 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.220619 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.220598 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.222901 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.222877 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:29.223058 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.223030 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:29.226667 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.226624 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.226742 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.226695 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:29.228785 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.228769 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.230912 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.230892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.232960 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.232945 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.235186 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.235169 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.237400 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.237382 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.239648 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.239616 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.241951 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.241933 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.248320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-systemd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.248409 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248328 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-var-lib-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.248409 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248344 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-log-socket\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.248409 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248361 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-system-cni-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.248559 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248444 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-hostroot\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.248559 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248481 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-multus-certs\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.248559 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.248559 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248541 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-netns\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248570 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-bin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248592 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-kubelet-config\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248626 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd9lf\" (UniqueName: \"kubernetes.io/projected/88200eac-4b64-4019-b733-335740bfe1e5-kube-api-access-pd9lf\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248710 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-cnibin\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248746 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88200eac-4b64-4019-b733-335740bfe1e5-hosts-file\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248768 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-registration-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248800 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-sys-fs\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248823 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-env-overrides\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.248862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248896 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248938 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.248986 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjgg\" (UniqueName: \"kubernetes.io/projected/ca0220e0-47de-45fb-bbb6-11ed27994448-kube-api-access-wgjgg\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovn-node-metrics-cert\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249036 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-kubelet\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-socket-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249082 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-systemd-units\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249096 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-ovn\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249129 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-node-log\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249190 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249162 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-system-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249204 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249231 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bd78\" (UniqueName: \"kubernetes.io/projected/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-kube-api-access-4bd78\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249251 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-etc-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249265 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249278 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-bin\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249290 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-etc-kubernetes\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249367 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a0eb70c-74c9-485b-a907-fa2830431a1d-agent-certs\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249389 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f94dde-4d2f-4969-b7ce-357282dc6529-host\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.249481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249454 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-kubelet\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249495 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249522 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-conf-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249550 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw657\" (UniqueName: \"kubernetes.io/projected/f0f94dde-4d2f-4969-b7ce-357282dc6529-kube-api-access-kw657\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249566 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249621 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-config\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249683 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-script-lib\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249713 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249741 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-multus\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249783 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4hnc\" (UniqueName: \"kubernetes.io/projected/772d8d93-5c01-4e16-832e-51dbcf7191bd-kube-api-access-f4hnc\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249843 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cnibin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249884 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-socket-dir-parent\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249919 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-netns\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249942 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-daemon-config\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249967 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.249993 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54cx9\" (UniqueName: \"kubernetes.io/projected/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-kube-api-access-54cx9\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.250578 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250027 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cni-binary-copy\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-device-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250079 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250147 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-slash\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-netd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250200 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a0eb70c-74c9-485b-a907-fa2830431a1d-konnectivity-ca\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250225 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f94dde-4d2f-4969-b7ce-357282dc6529-serviceca\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250276 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-dbus\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250302 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250325 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvcfs\" (UniqueName: \"kubernetes.io/projected/fba48a24-ec24-4b58-820c-c69010288372-kube-api-access-gvcfs\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250349 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-k8s-cni-cncf-io\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.251105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.250377 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88200eac-4b64-4019-b733-335740bfe1e5-tmp-dir\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.270903 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.270878 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.270992 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.270963 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:04:29.271054 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.271003 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:04:29.271234 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.271211 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:04:29.271316 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.271279 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.271316 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.271285 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ptgp6\"" Nov 03 00:04:29.271510 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.271495 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:04:29.275194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.273025 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.275194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.273527 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:04:29.275194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.273705 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:04:29.275194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.273845 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7949\"" Nov 03 00:04:29.275194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.273977 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:04:29.275194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.274098 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.281617 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.281601 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:04:29.281887 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.281868 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8glgl\"" Nov 03 00:04:29.283898 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.283883 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:04:29.283945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.283882 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:04:29.283945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.283932 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pkcxp\"" Nov 03 00:04:29.285860 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.285847 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.285860 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.285855 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6zhf\"" Nov 03 00:04:29.286153 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.286136 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.288574 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.288558 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.288574 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.288557 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:04:29.288689 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.288577 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.288689 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.288627 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f67qw\"" Nov 03 00:04:29.290544 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.290531 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.290939 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.290922 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scmmg\"" Nov 03 00:04:29.290939 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.290935 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:04:29.291038 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.290962 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.293251 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.293239 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.293306 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.293266 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.293359 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.293348 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nhzpn\"" Nov 03 00:04:29.317358 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.317334 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:29.317489 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.317473 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6628\"" Nov 03 00:04:29.317551 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.317493 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:29.317551 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.317502 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:04:29.322076 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.322059 2573 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 00:04:29.322145 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.322099 2573 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4lj7t" Nov 03 00:04:29.336194 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.336177 2573 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:04:29.351141 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351117 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-systemd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351229 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351166 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-systemd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351229 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-var-lib-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351229 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-log-socket\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351232 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-system-cni-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351240 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-var-lib-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351248 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-modprobe-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351285 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-log-socket\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351295 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-system-cni-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351333 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-run\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351361 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-hostroot\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.351395 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351386 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-multus-certs\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351411 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-multus-certs\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351431 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-hostroot\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351456 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351479 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-netns\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351498 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-conf\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351512 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-sys\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351527 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-tuned\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351543 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-netns\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351550 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-bin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351588 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-kubelet-config\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.351621 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351633 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-kubelet-config\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351687 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-bin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.351742 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.351700 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret podName:2ee7c64c-4945-469a-8d27-cfb702aabf50 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:29.851680243 +0000 UTC m=+3.035338222 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret") pod "global-pull-secret-syncer-cw8kk" (UID: "2ee7c64c-4945-469a-8d27-cfb702aabf50") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351768 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9lf\" (UniqueName: \"kubernetes.io/projected/88200eac-4b64-4019-b733-335740bfe1e5-kube-api-access-pd9lf\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-cnibin\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351809 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88200eac-4b64-4019-b733-335740bfe1e5-hosts-file\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-registration-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351852 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-sys-fs\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351867 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-env-overrides\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351894 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351893 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-cnibin\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351904 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88200eac-4b64-4019-b733-335740bfe1e5-hosts-file\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351914 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351927 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-registration-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351941 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-host\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.351953 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-sys-fs\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352030 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352069 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjgg\" (UniqueName: \"kubernetes.io/projected/ca0220e0-47de-45fb-bbb6-11ed27994448-kube-api-access-wgjgg\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352086 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.352320 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352094 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovn-node-metrics-cert\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352120 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352140 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-lib-modules\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352162 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-kubelet\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352187 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-socket-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-systemd-units\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352253 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-kubelet\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352312 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-ovn\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352330 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-systemd-units\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352371 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-socket-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352377 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-node-log\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352421 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-system-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352433 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-node-log\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352374 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-ovn\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352478 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bd78\" (UniqueName: \"kubernetes.io/projected/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-kube-api-access-4bd78\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352507 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-tmp\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352512 2573 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:04:29.353065 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352531 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-etc-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352551 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352592 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352601 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-bin\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352628 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-systemd\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352659 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-env-overrides\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352673 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-etc-kubernetes\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352697 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a0eb70c-74c9-485b-a907-fa2830431a1d-agent-certs\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352516 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-system-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f94dde-4d2f-4969-b7ce-357282dc6529-host\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352781 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352796 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352804 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-kubelet\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352824 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-bin\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352852 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-etc-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352930 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f94dde-4d2f-4969-b7ce-357282dc6529-host\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.353840 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352931 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-etc-kubernetes\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352938 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.352970 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-kubelet\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353020 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353024 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-iptables-alerter-script\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353066 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-host-slash\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353124 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysconfig\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353178 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8tz\" (UniqueName: \"kubernetes.io/projected/deb16adf-e150-4c31-8b3f-6509c0b919e3-kube-api-access-5r8tz\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353216 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-conf-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-conf-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353267 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw657\" (UniqueName: \"kubernetes.io/projected/f0f94dde-4d2f-4969-b7ce-357282dc6529-kube-api-access-kw657\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353307 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-config\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353353 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-script-lib\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353388 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353418 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-multus\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353471 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hnc\" (UniqueName: \"kubernetes.io/projected/772d8d93-5c01-4e16-832e-51dbcf7191bd-kube-api-access-f4hnc\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.354647 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353524 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cnibin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-socket-dir-parent\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353580 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-netns\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353606 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-daemon-config\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353632 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353659 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54cx9\" (UniqueName: \"kubernetes.io/projected/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-kube-api-access-54cx9\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353702 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353729 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-779m4\" (UniqueName: \"kubernetes.io/projected/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-kube-api-access-779m4\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353764 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cni-binary-copy\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353790 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-device-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353800 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353823 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353845 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-multus\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353850 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cnibin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353848 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-slash\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353900 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-netd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353912 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.355299 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353933 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-kubernetes\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353962 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-var-lib-kubelet\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-netns\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353982 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-slash\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353986 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a0eb70c-74c9-485b-a907-fa2830431a1d-konnectivity-ca\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354017 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-config\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354035 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-device-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.353795 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-socket-dir-parent\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f94dde-4d2f-4969-b7ce-357282dc6529-serviceca\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354083 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-netd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354092 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-dbus\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354121 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354127 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354161 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcfs\" (UniqueName: \"kubernetes.io/projected/fba48a24-ec24-4b58-820c-c69010288372-kube-api-access-gvcfs\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-k8s-cni-cncf-io\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88200eac-4b64-4019-b733-335740bfe1e5-tmp-dir\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.354257 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:29.355801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354303 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-k8s-cni-cncf-io\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.354480 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:04:29.854459904 +0000 UTC m=+3.038117868 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354505 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cni-binary-copy\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354612 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88200eac-4b64-4019-b733-335740bfe1e5-tmp-dir\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354656 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354671 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a0eb70c-74c9-485b-a907-fa2830431a1d-konnectivity-ca\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.354728 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-dbus\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.355012 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f94dde-4d2f-4969-b7ce-357282dc6529-serviceca\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.355938 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-daemon-config\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.356047 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-script-lib\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.356241 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovn-node-metrics-cert\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.356457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.356404 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a0eb70c-74c9-485b-a907-fa2830431a1d-agent-certs\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.413321 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.413248 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:29.413321 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.413270 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:29.413321 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.413283 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qmqm4 for pod openshift-network-diagnostics/network-check-target-gbt9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:29.413507 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.413371 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4 podName:6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:29.913352196 +0000 UTC m=+3.097010176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-qmqm4" (UniqueName: "kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4") pod "network-check-target-gbt9l" (UID: "6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:29.415373 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.415350 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9lf\" (UniqueName: \"kubernetes.io/projected/88200eac-4b64-4019-b733-335740bfe1e5-kube-api-access-pd9lf\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.415632 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.415615 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bd78\" (UniqueName: \"kubernetes.io/projected/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-kube-api-access-4bd78\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.415689 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.415674 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjgg\" (UniqueName: \"kubernetes.io/projected/ca0220e0-47de-45fb-bbb6-11ed27994448-kube-api-access-wgjgg\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.419080 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.419052 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hnc\" (UniqueName: \"kubernetes.io/projected/772d8d93-5c01-4e16-832e-51dbcf7191bd-kube-api-access-f4hnc\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.419208 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.419184 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw657\" (UniqueName: \"kubernetes.io/projected/f0f94dde-4d2f-4969-b7ce-357282dc6529-kube-api-access-kw657\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.419550 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.419534 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcfs\" (UniqueName: \"kubernetes.io/projected/fba48a24-ec24-4b58-820c-c69010288372-kube-api-access-gvcfs\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.419685 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.419669 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54cx9\" (UniqueName: \"kubernetes.io/projected/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-kube-api-access-54cx9\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.454571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454551 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779m4\" (UniqueName: \"kubernetes.io/projected/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-kube-api-access-779m4\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.454620 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454577 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-kubernetes\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454620 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454592 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-var-lib-kubelet\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454620 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454610 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-modprobe-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454734 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454674 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-kubernetes\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454766 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454731 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-run\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454820 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454767 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-conf\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454820 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454791 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-sys\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454820 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454807 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-tuned\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454832 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-modprobe-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454840 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-run\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454848 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-var-lib-kubelet\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-host\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-sys\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454875 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-host\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454898 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454874 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-conf\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454926 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-lib-modules\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454956 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-tmp\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.454983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.454974 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-systemd\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455006 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-systemd\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455009 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-iptables-alerter-script\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455018 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-lib-modules\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455046 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-host-slash\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455064 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysconfig\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455087 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8tz\" (UniqueName: \"kubernetes.io/projected/deb16adf-e150-4c31-8b3f-6509c0b919e3-kube-api-access-5r8tz\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455106 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-host-slash\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.455468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455151 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysconfig\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.455812 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.455791 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-iptables-alerter-script\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.456919 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.456900 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-tmp\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.457457 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.457439 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-tuned\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.516040 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.516014 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779m4\" (UniqueName: \"kubernetes.io/projected/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-kube-api-access-779m4\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.516148 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.516129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8tz\" (UniqueName: \"kubernetes.io/projected/deb16adf-e150-4c31-8b3f-6509c0b919e3-kube-api-access-5r8tz\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.545753 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.545726 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wbjwv" Nov 03 00:04:29.551157 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.551132 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:29.555431 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.555411 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d665bcd_2a5f_45c8_802a_370fc4a4aefd.slice/crio-707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc WatchSource:0}: Error finding container 707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc: Status 404 returned error can't find the container with id 707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc Nov 03 00:04:29.558808 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.558787 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4vplk" Nov 03 00:04:29.559493 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.559474 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:29.559900 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.559879 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0eb70c_74c9_485b_a907_fa2830431a1d.slice/crio-fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3 WatchSource:0}: Error finding container fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3: Status 404 returned error can't find the container with id fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3 Nov 03 00:04:29.567991 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.567977 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:04:29.568349 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.568326 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88200eac_4b64_4019_b733_335740bfe1e5.slice/crio-df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3 WatchSource:0}: Error finding container df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3: Status 404 returned error can't find the container with id df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3 Nov 03 00:04:29.571875 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.571858 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f3669a_ce7d_4df7_a9d6_7da101fe3fd4.slice/crio-a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb WatchSource:0}: Error finding container a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb: Status 404 returned error can't find the container with id a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb Nov 03 00:04:29.572425 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.572407 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:04:29.575910 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.575892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:04:29.577374 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.577351 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f94dde_4d2f_4969_b7ce_357282dc6529.slice/crio-fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201 WatchSource:0}: Error finding container fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201: Status 404 returned error can't find the container with id fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201 Nov 03 00:04:29.577883 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.577867 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:04:29.580721 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.580686 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:04:29.582827 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.582255 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca0220e0_47de_45fb_bbb6_11ed27994448.slice/crio-b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe WatchSource:0}: Error finding container b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe: Status 404 returned error can't find the container with id b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe Nov 03 00:04:29.585943 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.585910 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c44bcac_ee48_48af_b5b4_99f2ce0e81d6.slice/crio-ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc WatchSource:0}: Error finding container ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc: Status 404 returned error can't find the container with id ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc Nov 03 00:04:29.589893 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.589875 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba48a24_ec24_4b58_820c_c69010288372.slice/crio-2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40 WatchSource:0}: Error finding container 2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40: Status 404 returned error can't find the container with id 2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40 Nov 03 00:04:29.592956 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:29.592933 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeb16adf_e150_4c31_8b3f_6509c0b919e3.slice/crio-0d9347071dacb553d383ea0d844ee2abc26acedc928c763a464d0d1774098e3e WatchSource:0}: Error finding container 0d9347071dacb553d383ea0d844ee2abc26acedc928c763a464d0d1774098e3e: Status 404 returned error can't find the container with id 0d9347071dacb553d383ea0d844ee2abc26acedc928c763a464d0d1774098e3e Nov 03 00:04:29.858009 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.857945 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:29.858108 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.858028 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:29.858108 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.858055 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:29.858196 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.858114 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret podName:2ee7c64c-4945-469a-8d27-cfb702aabf50 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:30.858098189 +0000 UTC m=+4.041756149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret") pod "global-pull-secret-syncer-cw8kk" (UID: "2ee7c64c-4945-469a-8d27-cfb702aabf50") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:29.858196 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.858115 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:29.858196 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.858150 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:04:30.858141372 +0000 UTC m=+4.041799345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:29.958828 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:29.958796 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:29.958977 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.958960 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:29.959037 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.958983 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:29.959037 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.958993 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qmqm4 for pod openshift-network-diagnostics/network-check-target-gbt9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:29.959095 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:29.959047 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4 podName:6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:30.959030665 +0000 UTC m=+4.142688638 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmqm4" (UniqueName: "kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4") pod "network-check-target-gbt9l" (UID: "6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:30.213326 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.213223 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerStarted","Data":"2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40"} Nov 03 00:04:30.214734 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.214696 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t4jtn" event={"ID":"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6","Type":"ContainerStarted","Data":"ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc"} Nov 03 00:04:30.216550 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.216522 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" event={"ID":"ca0220e0-47de-45fb-bbb6-11ed27994448","Type":"ContainerStarted","Data":"b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe"} Nov 03 00:04:30.217980 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.217951 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb"} Nov 03 00:04:30.220701 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.220674 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z8d7r" event={"ID":"1a0eb70c-74c9-485b-a907-fa2830431a1d","Type":"ContainerStarted","Data":"fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3"} Nov 03 00:04:30.222697 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.222655 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfw72" event={"ID":"deb16adf-e150-4c31-8b3f-6509c0b919e3","Type":"ContainerStarted","Data":"0d9347071dacb553d383ea0d844ee2abc26acedc928c763a464d0d1774098e3e"} Nov 03 00:04:30.225859 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.225786 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dgbsh" event={"ID":"f0f94dde-4d2f-4969-b7ce-357282dc6529","Type":"ContainerStarted","Data":"fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201"} Nov 03 00:04:30.227184 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.227138 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vplk" event={"ID":"88200eac-4b64-4019-b733-335740bfe1e5","Type":"ContainerStarted","Data":"df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3"} Nov 03 00:04:30.228655 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.228616 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbjwv" event={"ID":"7d665bcd-2a5f-45c8-802a-370fc4a4aefd","Type":"ContainerStarted","Data":"707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc"} Nov 03 00:04:30.325326 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.325269 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:29 +0000 UTC" deadline="2028-06-18 17:16:39.736599361 +0000 UTC" Nov 03 00:04:30.325326 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.325321 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14249h12m9.41128199s" Nov 03 00:04:30.865040 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.864999 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:30.865229 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.865097 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:30.865307 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.865233 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:30.865307 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.865299 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:04:32.865279325 +0000 UTC m=+6.048937283 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:30.865693 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.865675 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:30.865792 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.865748 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret podName:2ee7c64c-4945-469a-8d27-cfb702aabf50 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:32.865732078 +0000 UTC m=+6.049390041 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret") pod "global-pull-secret-syncer-cw8kk" (UID: "2ee7c64c-4945-469a-8d27-cfb702aabf50") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:30.966277 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:30.966001 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:30.966277 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.966144 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:30.966277 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.966164 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:30.966277 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.966186 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qmqm4 for pod openshift-network-diagnostics/network-check-target-gbt9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:30.966277 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:30.966239 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4 podName:6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:32.966220365 +0000 UTC m=+6.149878345 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmqm4" (UniqueName: "kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4") pod "network-check-target-gbt9l" (UID: "6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:31.206981 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:31.206849 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:31.207145 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:31.207000 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:31.207410 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:31.207387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:31.207520 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:31.207499 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:31.207585 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:31.207561 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:31.207661 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:31.207641 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:31.325802 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:31.325734 2573 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:29 +0000 UTC" deadline="2028-05-30 00:47:27.609070499 +0000 UTC" Nov 03 00:04:31.325802 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:31.325771 2573 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13776h42m56.283304242s" Nov 03 00:04:32.882349 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:32.882249 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:32.882349 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:32.882337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:32.882892 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.882457 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:32.882892 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.882518 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:04:36.882498762 +0000 UTC m=+10.066156735 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:32.883009 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.882935 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:32.883009 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.882990 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret podName:2ee7c64c-4945-469a-8d27-cfb702aabf50 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:36.882974995 +0000 UTC m=+10.066632967 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret") pod "global-pull-secret-syncer-cw8kk" (UID: "2ee7c64c-4945-469a-8d27-cfb702aabf50") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:32.983724 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:32.983685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:32.983871 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.983838 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:32.983871 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.983852 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:32.983871 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.983864 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qmqm4 for pod openshift-network-diagnostics/network-check-target-gbt9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:32.984037 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:32.983910 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4 podName:6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:36.9838986 +0000 UTC m=+10.167556572 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmqm4" (UniqueName: "kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4") pod "network-check-target-gbt9l" (UID: "6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:33.209796 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:33.208148 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:33.209796 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:33.208522 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:33.209796 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:33.208927 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:33.209796 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:33.209018 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:33.209796 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:33.209081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:33.209796 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:33.209158 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:33.255739 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:33.254069 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"6b8cde6519a41e629e9c7fa53310b7cb","Type":"ContainerStarted","Data":"f8a9c81cc7d435376a727ff1bb5ad739b979c80da53d322f9ae82960f6ed2c17"} Nov 03 00:04:34.273901 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:34.273861 2573 generic.go:356] "Generic (PLEG): container finished" podID="2a0942cf01c78617561368216297af20" containerID="c12b9ebc180ac2019d6b8125d82390dc18bd5ee28f4f81c5c86edffcabbe6c1e" exitCode=0 Nov 03 00:04:34.274342 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:34.273963 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"2a0942cf01c78617561368216297af20","Type":"ContainerDied","Data":"c12b9ebc180ac2019d6b8125d82390dc18bd5ee28f4f81c5c86edffcabbe6c1e"} Nov 03 00:04:34.284778 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:34.283915 2573 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="df15ea5a936fa03abb7788dce2cf1076737fd27aba9b4bb705508baac10a3e1e" exitCode=0 Nov 03 00:04:34.284778 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:34.284713 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerDied","Data":"df15ea5a936fa03abb7788dce2cf1076737fd27aba9b4bb705508baac10a3e1e"} Nov 03 00:04:34.398249 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:34.397911 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" podStartSLOduration=5.397892133 podStartE2EDuration="5.397892133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:33.37659163 +0000 UTC m=+6.560249613" watchObservedRunningTime="2026-11-03 00:04:34.397892133 +0000 UTC m=+7.581550117" Nov 03 00:04:35.216932 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:35.216878 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:35.217127 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:35.217090 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:35.218020 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:35.217608 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:35.218020 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:35.217775 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:35.218020 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:35.217851 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:35.218020 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:35.217972 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:35.289695 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:35.289639 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"2a0942cf01c78617561368216297af20","Type":"ContainerStarted","Data":"326f5e029ba1827904071effc62f2de863d27c8a0af7af7d672086c32c0f8db2"} Nov 03 00:04:35.292158 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:35.292129 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-z8d7r" event={"ID":"1a0eb70c-74c9-485b-a907-fa2830431a1d","Type":"ContainerStarted","Data":"758deebffa06fd7df51ccf8a51480208739f8d65d32517b37483c1b8d860dd48"} Nov 03 00:04:35.412276 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:35.412215 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" podStartSLOduration=6.412195797 podStartE2EDuration="6.412195797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:35.411734928 +0000 UTC m=+8.595392908" watchObservedRunningTime="2026-11-03 00:04:35.412195797 +0000 UTC m=+8.595853780" Nov 03 00:04:36.299664 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:36.299570 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-dgbsh" event={"ID":"f0f94dde-4d2f-4969-b7ce-357282dc6529","Type":"ContainerStarted","Data":"41c4413f42fef1620af2fc6b1ab1aa57b801d84134415153ec1722521419db0c"} Nov 03 00:04:36.412009 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:36.411955 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-z8d7r" podStartSLOduration=3.791494649 podStartE2EDuration="8.411934328s" totalImagesPullingTime="4.620439679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:35.532975278 +0000 UTC m=+8.716633281" watchObservedRunningTime="2026-11-03 00:04:36.411934328 +0000 UTC m=+9.595592307" Nov 03 00:04:36.918365 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:36.918337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:36.918482 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:36.918408 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:36.918546 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:36.918523 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:36.918596 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:36.918578 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:04:44.918557979 +0000 UTC m=+18.102215957 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:36.918702 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:36.918688 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:36.918755 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:36.918746 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret podName:2ee7c64c-4945-469a-8d27-cfb702aabf50 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:44.9187231 +0000 UTC m=+18.102381059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret") pod "global-pull-secret-syncer-cw8kk" (UID: "2ee7c64c-4945-469a-8d27-cfb702aabf50") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:37.018970 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:37.018886 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:37.019134 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.019023 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:37.019134 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.019041 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:37.019134 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.019052 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qmqm4 for pod openshift-network-diagnostics/network-check-target-gbt9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:37.019134 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.019101 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4 podName:6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:45.019083473 +0000 UTC m=+18.202741429 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmqm4" (UniqueName: "kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4") pod "network-check-target-gbt9l" (UID: "6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:37.206294 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:37.206257 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:37.206471 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.206395 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:37.207267 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:37.207239 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:37.207386 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.207330 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:37.207386 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:37.207375 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:37.207495 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:37.207445 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:37.305809 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:37.305729 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" event={"ID":"ca0220e0-47de-45fb-bbb6-11ed27994448","Type":"ContainerStarted","Data":"50d54f1e057a26c0e0000682e819a97ce7d21fc0a808035190c2ec54da496813"} Nov 03 00:04:38.309907 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.309828 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4vplk" event={"ID":"88200eac-4b64-4019-b733-335740bfe1e5","Type":"ContainerStarted","Data":"2d209bda28c088401ca7a9026f55cbeef4e9de6bbfda6e2ef21b74b60d132ef8"} Nov 03 00:04:38.312529 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.312497 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wbjwv" event={"ID":"7d665bcd-2a5f-45c8-802a-370fc4a4aefd","Type":"ContainerStarted","Data":"90612fa460d7cb878266eb45b1bf3fafc0b2055e476ead3ee5dfdace923b2ea9"} Nov 03 00:04:38.421307 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.421260 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-dgbsh" podStartSLOduration=4.037136439 podStartE2EDuration="10.421246858s" totalImagesPullingTime="6.384110419s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:36.414179535 +0000 UTC m=+9.597837512" watchObservedRunningTime="2026-11-03 00:04:38.421246858 +0000 UTC m=+11.604904849" Nov 03 00:04:38.421471 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.421414 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4vplk" podStartSLOduration=2.17659013 podStartE2EDuration="10.421406971s" totalImagesPullingTime="8.244816841s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:38.420872717 +0000 UTC m=+11.604530695" watchObservedRunningTime="2026-11-03 00:04:38.421406971 +0000 UTC m=+11.605064950" Nov 03 00:04:38.585206 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.585101 2573 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:04:38.796950 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.796920 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:38.797810 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.797788 2573 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:38.909214 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:38.909105 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wbjwv" podStartSLOduration=2.874546622 podStartE2EDuration="10.909085783s" totalImagesPullingTime="8.034539161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:38.537289414 +0000 UTC m=+11.720947393" watchObservedRunningTime="2026-11-03 00:04:38.909085783 +0000 UTC m=+12.092743761" Nov 03 00:04:39.208667 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.208572 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:39.208806 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:39.208714 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:39.209100 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.209074 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:39.209198 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:39.209170 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:39.209245 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.209211 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:39.209277 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:39.209251 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:39.220432 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.220343 2573 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:04:38.585121376Z","UUID":"a1683636-d215-4931-904e-f87ffd5d0cb4","Handler":null,"Name":"","Endpoint":""} Nov 03 00:04:39.222434 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.222108 2573 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:04:39.222434 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.222138 2573 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:04:39.316967 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.316927 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t4jtn" event={"ID":"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6","Type":"ContainerStarted","Data":"1f85b3ecd9f8c7d02c05c20cf43f71bfa126f3bfe4b30ce2b287596bfcf1a87a"} Nov 03 00:04:39.319202 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:39.319169 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" event={"ID":"ca0220e0-47de-45fb-bbb6-11ed27994448","Type":"ContainerStarted","Data":"9d8547bcbd15b3934ace9fbe3bdb2391cb9daf43144c0372467fff6478198fcd"} Nov 03 00:04:40.323292 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:40.323041 2573 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="5e41d8846367d47a914ef93051a8be115fe03b6ac3a632672a7d2be9d7aa4969" exitCode=0 Nov 03 00:04:40.323652 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:40.323124 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerDied","Data":"5e41d8846367d47a914ef93051a8be115fe03b6ac3a632672a7d2be9d7aa4969"} Nov 03 00:04:40.530127 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:40.530088 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t4jtn" podStartSLOduration=4.154320391 podStartE2EDuration="12.530071921s" totalImagesPullingTime="8.37575153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:39.428473793 +0000 UTC m=+12.612131798" watchObservedRunningTime="2026-11-03 00:04:40.530071921 +0000 UTC m=+13.713729899" Nov 03 00:04:41.205913 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.205884 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:41.205913 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.205911 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:41.206139 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.205886 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:41.206139 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:41.206020 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:41.206139 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:41.206099 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:41.206277 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:41.206216 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:41.327126 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.327091 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" event={"ID":"ca0220e0-47de-45fb-bbb6-11ed27994448","Type":"ContainerStarted","Data":"c22ada9fff18e87e59e6318afd8304285128f7d330a7e494ea3d12eccf6dd14e"} Nov 03 00:04:41.328761 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.328734 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lfw72" event={"ID":"deb16adf-e150-4c31-8b3f-6509c0b919e3","Type":"ContainerStarted","Data":"c01b3c45b21f3ee020fbc8cf3c2a831d6836276816d49d6200736401ce062151"} Nov 03 00:04:41.441035 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.440988 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" podStartSLOduration=3.03361488 podStartE2EDuration="13.440974116s" totalImagesPullingTime="10.407359236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:41.440457784 +0000 UTC m=+14.624115764" watchObservedRunningTime="2026-11-03 00:04:41.440974116 +0000 UTC m=+14.624632093" Nov 03 00:04:41.554982 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:41.554889 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lfw72" podStartSLOduration=2.674032138 podStartE2EDuration="13.554880108s" totalImagesPullingTime="10.88084797s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:41.554472125 +0000 UTC m=+14.738130105" watchObservedRunningTime="2026-11-03 00:04:41.554880108 +0000 UTC m=+14.738538085" Nov 03 00:04:42.333137 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:42.333098 2573 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="6c017b81f0b7feb3d6510550085eb69df155e76c256d9d35c9448bd9173b599c" exitCode=0 Nov 03 00:04:42.333759 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:42.333186 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerDied","Data":"6c017b81f0b7feb3d6510550085eb69df155e76c256d9d35c9448bd9173b599c"} Nov 03 00:04:43.206197 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:43.206159 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:43.206384 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:43.206164 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:43.206384 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:43.206300 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:43.206384 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:43.206279 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:43.206526 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:43.206416 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:43.206526 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:43.206503 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:44.339048 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:44.338870 2573 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="70281a9da8f50d744623c3b321bd266a2ab35a5f3d33b1cfafc22bb73ad098e9" exitCode=0 Nov 03 00:04:44.339048 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:44.338944 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerDied","Data":"70281a9da8f50d744623c3b321bd266a2ab35a5f3d33b1cfafc22bb73ad098e9"} Nov 03 00:04:44.981011 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:44.980942 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:44.981011 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:44.981008 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:44.981203 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:44.981066 2573 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:44.981203 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:44.981099 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:44.981203 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:44.981115 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret podName:2ee7c64c-4945-469a-8d27-cfb702aabf50 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.98110214 +0000 UTC m=+34.164760096 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret") pod "global-pull-secret-syncer-cw8kk" (UID: "2ee7c64c-4945-469a-8d27-cfb702aabf50") : object "kube-system"/"original-pull-secret" not registered Nov 03 00:04:44.981203 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:44.981141 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.981127547 +0000 UTC m=+34.164785503 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 00:04:45.081935 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.081907 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:45.082052 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.081986 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 00:04:45.082052 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.081996 2573 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 00:04:45.082052 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.082005 2573 projected.go:196] Error preparing data for projected volume kube-api-access-qmqm4 for pod openshift-network-diagnostics/network-check-target-gbt9l: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:45.082052 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.082037 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4 podName:6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:01.082028303 +0000 UTC m=+34.265686260 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-qmqm4" (UniqueName: "kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4") pod "network-check-target-gbt9l" (UID: "6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 00:04:45.205839 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.205816 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:45.205937 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.205890 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:45.205937 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.205815 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:45.205937 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.205913 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:45.206100 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.205958 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:45.206100 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:45.206019 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:45.344860 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.344660 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"181c9197953516c57307447ea38eaa6c2e12fbe8276134dde2bd95724a7248b8"} Nov 03 00:04:45.345331 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.344863 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"9c347e3139d0981ff40b00c7217b5c3c2b9cd40eea9489283804ec001321d37f"} Nov 03 00:04:45.345331 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.344907 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"7df1ceaef434d0b46b444248a24f92b54f1e189213034120894167b3664eacf5"} Nov 03 00:04:45.345331 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.344936 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"86405b915018bd01a76febd3f8130a65cd6a564c2644d3e87b16ac4b0b0974ad"} Nov 03 00:04:45.345331 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.344962 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"6ef37fb591c96297f8f1b1aa80d1cc9e814eae8a517bf6b6ad4994e9f9db0bed"} Nov 03 00:04:45.345331 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:45.344994 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"2ee32ad20b8685b06ddcc1182924c63a4cb031a639305be4a9ce3f3c98c264e5"} Nov 03 00:04:47.205862 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:47.205829 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:47.206309 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:47.205937 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:47.206309 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:47.205976 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:47.206309 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:47.206054 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:47.206309 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:47.206116 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:47.206309 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:47.206214 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:47.351649 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:47.351609 2573 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="ec317e1278e616a29ca02b8f96307a717ffa61c0387b4f7afbb8ed337f244465" exitCode=0 Nov 03 00:04:47.351792 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:47.351662 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerDied","Data":"ec317e1278e616a29ca02b8f96307a717ffa61c0387b4f7afbb8ed337f244465"} Nov 03 00:04:47.354872 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:47.354853 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"ef19466dcef6d007d733c2e89bcfb5913f885cadf6daa06f35cf6034d5ac8a09"} Nov 03 00:04:48.359746 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:48.359713 2573 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="f7fd299f32d310da05fda990e7b2aff353e4a783341a71bc65ec2ca966448de2" exitCode=0 Nov 03 00:04:48.360164 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:48.359769 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerDied","Data":"f7fd299f32d310da05fda990e7b2aff353e4a783341a71bc65ec2ca966448de2"} Nov 03 00:04:49.208211 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.208188 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:49.208351 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.208217 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:49.208351 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.208214 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:49.208351 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:49.208301 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:49.208457 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:49.208371 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:49.208457 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:49.208411 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:49.364717 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.364693 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" event={"ID":"fba48a24-ec24-4b58-820c-c69010288372","Type":"ContainerStarted","Data":"3aed4094bbabb7350f5fd02c7bd95ea1f3655bd69b4b06e107951e58fd8fa893"} Nov 03 00:04:49.367986 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.367965 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" event={"ID":"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4","Type":"ContainerStarted","Data":"264684179d041b32ccde7300add19e58aa7e8511823ae74938d1d61b22f8559d"} Nov 03 00:04:49.368217 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.368195 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:49.368340 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.368250 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:49.368340 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.368291 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:49.383117 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.383099 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:49.383272 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.383258 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:04:49.480942 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.480863 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5mqcz" podStartSLOduration=8.084836431 podStartE2EDuration="21.480851815s" totalImagesPullingTime="13.396015384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:49.480747162 +0000 UTC m=+22.664405138" watchObservedRunningTime="2026-11-03 00:04:49.480851815 +0000 UTC m=+22.664509792" Nov 03 00:04:49.602423 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:49.602385 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" podStartSLOduration=6.884906458 podStartE2EDuration="21.602374641s" totalImagesPullingTime="14.717468183s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:49.601972069 +0000 UTC m=+22.785630046" watchObservedRunningTime="2026-11-03 00:04:49.602374641 +0000 UTC m=+22.786032618" Nov 03 00:04:51.209122 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.208944 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:51.209586 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.209035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:51.209586 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:51.209195 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:51.209586 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.209046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:51.209586 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:51.209316 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:51.209586 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:51.209394 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:51.416008 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.415975 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cw8kk"] Nov 03 00:04:51.416191 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.416081 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:51.416191 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:51.416168 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:51.416475 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.416445 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lcd5z"] Nov 03 00:04:51.416623 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.416548 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:51.416695 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:51.416653 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:51.425705 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.425683 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbt9l"] Nov 03 00:04:51.425793 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:51.425760 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:51.425845 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:51.425830 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:53.208253 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:53.208144 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:53.208253 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:53.208237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:53.208751 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:53.208339 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:53.208751 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:53.208363 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:53.208751 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:53.208420 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:53.208751 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:53.208450 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:55.208353 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:55.208290 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:55.208703 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:55.208387 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:55.208703 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:55.208399 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cw8kk" podUID="2ee7c64c-4945-469a-8d27-cfb702aabf50" Nov 03 00:04:55.208703 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:55.208467 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-lcd5z" podUID="772d8d93-5c01-4e16-832e-51dbcf7191bd" Nov 03 00:04:55.208703 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:55.208491 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:55.208703 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:55.208594 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gbt9l" podUID="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" Nov 03 00:04:56.605518 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.605492 2573 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeReady" Nov 03 00:04:56.605985 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.605591 2573 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 00:04:56.719007 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.718979 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kzvvs"] Nov 03 00:04:56.721235 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.721220 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mvcq4"] Nov 03 00:04:56.721361 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.721344 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:04:56.723835 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.723812 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gtptx"] Nov 03 00:04:56.723957 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.723917 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:04:56.726459 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.726438 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm"] Nov 03 00:04:56.726567 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.726551 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:04:56.730777 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.730757 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.765421 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.765401 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thvvt\" (UniqueName: \"kubernetes.io/projected/d7e0f3c7-a695-4b00-8178-1955b1159469-kube-api-access-thvvt\") pod \"kas-connection-checker-685895c7c9-kzvvs\" (UID: \"d7e0f3c7-a695-4b00-8178-1955b1159469\") " pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:04:56.768819 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.768802 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6"] Nov 03 00:04:56.771459 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.771443 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w"] Nov 03 00:04:56.771559 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.771543 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.774128 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.774108 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mdw87"] Nov 03 00:04:56.774223 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.774212 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.776717 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.776698 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cdfnj"] Nov 03 00:04:56.776810 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.776798 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.779721 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.779386 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jm2js"] Nov 03 00:04:56.779721 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.779682 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.782894 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.782876 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-pruner-29894400-7th8w"] Nov 03 00:04:56.783062 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.782984 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:56.785854 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.785838 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cd89fb64c-8bxkm"] Nov 03 00:04:56.785942 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.785928 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:56.788527 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.788510 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-767574d6c-vss4j"] Nov 03 00:04:56.788613 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.788601 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.791374 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.791355 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp"] Nov 03 00:04:56.791455 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.791445 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.794802 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.794784 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:04:56.818412 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.818384 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.818757 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.818731 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.819933 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.819911 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c595ff78b-zkrdt"] Nov 03 00:04:56.825143 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.824751 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6"] Nov 03 00:04:56.825143 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.825131 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.828549 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.828531 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx"] Nov 03 00:04:56.828952 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.828935 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:56.829130 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.829115 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 00:04:56.832776 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.832756 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-n2pqm"] Nov 03 00:04:56.832974 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.832955 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:04:56.835909 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.835874 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-xs77t"] Nov 03 00:04:56.839928 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.839910 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:56.840961 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.840944 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mvcq4"] Nov 03 00:04:56.841047 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.840979 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gtptx"] Nov 03 00:04:56.841047 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.840997 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kzvvs"] Nov 03 00:04:56.841047 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.841015 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jm2js"] Nov 03 00:04:56.841047 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.841034 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w"] Nov 03 00:04:56.841232 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.841053 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-7th8w"] Nov 03 00:04:56.841232 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.841071 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mdw87"] Nov 03 00:04:56.841232 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.841091 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6"] Nov 03 00:04:56.841232 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.841168 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:56.847268 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.847254 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-767574d6c-vss4j"] Nov 03 00:04:56.863067 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.863049 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:56.863158 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.863139 2573 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:04:56.863606 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.863593 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:04:56.865710 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865693 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae7af53a-1838-409e-b5f3-6aa5c688dd55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:56.865789 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865716 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.865789 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865735 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsx2q\" (UniqueName: \"kubernetes.io/projected/35ce84ec-29fd-49ab-9c7a-092e99f093b8-kube-api-access-jsx2q\") pod \"kas-connection-checker-685895c7c9-gtptx\" (UID: \"35ce84ec-29fd-49ab-9c7a-092e99f093b8\") " pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:04:56.865789 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865751 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:56.865789 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865769 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.865978 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865819 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-trusted-ca\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.865978 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865892 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f58b544f-c636-4c93-9e1c-9a817590d307-serving-cert\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.865978 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865931 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvvt\" (UniqueName: \"kubernetes.io/projected/d7e0f3c7-a695-4b00-8178-1955b1159469-kube-api-access-thvvt\") pod \"kas-connection-checker-685895c7c9-kzvvs\" (UID: \"d7e0f3c7-a695-4b00-8178-1955b1159469\") " pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:04:56.866104 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.865989 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-default-certificate\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.866104 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866018 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.866104 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866037 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trkh5\" (UniqueName: \"kubernetes.io/projected/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-kube-api-access-trkh5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.866104 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866057 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.866104 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866071 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-stats-auth\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.866104 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866091 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866107 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwkv\" (UniqueName: \"kubernetes.io/projected/f58b544f-c636-4c93-9e1c-9a817590d307-kube-api-access-rpwkv\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866121 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866162 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c5244e9-0e47-4a6d-a27d-24ce8455a537-serving-cert\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866237 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866270 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtb2k\" (UniqueName: \"kubernetes.io/projected/0bbc3b7e-1648-4984-b4da-673a033aca22-kube-api-access-dtb2k\") pod \"volume-data-source-validator-58c9f5f46d-lnvpp\" (UID: \"0bbc3b7e-1648-4984-b4da-673a033aca22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866350 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866387 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qj2\" (UniqueName: \"kubernetes.io/projected/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-kube-api-access-n4qj2\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866413 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866449 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-service-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866524 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866555 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfhs5\" (UniqueName: \"kubernetes.io/projected/82a02683-66a4-43a5-9d32-c53f72d1f613-kube-api-access-vfhs5\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.866571 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866577 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-config\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866623 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866699 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866760 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866784 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866814 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6jzx\" (UniqueName: \"kubernetes.io/projected/1edd18e2-9aa6-4708-98e9-773c0a38a368-kube-api-access-j6jzx\") pod \"kas-connection-checker-685895c7c9-mvcq4\" (UID: \"1edd18e2-9aa6-4708-98e9-773c0a38a368\") " pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866867 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.866956 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867009 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-tmp\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867062 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-447n8\" (UniqueName: \"kubernetes.io/projected/7c5244e9-0e47-4a6d-a27d-24ce8455a537-kube-api-access-447n8\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867407 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-snapshots\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867423 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867445 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t5vr\" (UniqueName: \"kubernetes.io/projected/31e371fb-f718-43d0-bf0f-d909838ac2d0-kube-api-access-9t5vr\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.867468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.867476 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.880462 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880445 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.880462 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880456 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 00:04:56.880849 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880702 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 00:04:56.880849 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880712 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Nov 03 00:04:56.880849 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880712 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.880849 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880739 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.880849 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880766 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.880849 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880807 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.881062 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880925 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.881062 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.880928 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.881208 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881192 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.881321 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881307 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 00:04:56.881386 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881332 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.881386 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881352 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 00:04:56.881459 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881386 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 00:04:56.881459 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881385 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.881532 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881488 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.881532 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881517 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.881604 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881531 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.881604 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.881553 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 00:04:56.896339 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.896322 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.930963 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.930942 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 00:04:56.937214 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.937191 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx"] Nov 03 00:04:56.946031 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.946016 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 00:04:56.946448 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.946433 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm"] Nov 03 00:04:56.968328 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968308 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.968375 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968331 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-stats-auth\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.968375 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968347 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:56.968375 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968364 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwkv\" (UniqueName: \"kubernetes.io/projected/f58b544f-c636-4c93-9e1c-9a817590d307-kube-api-access-rpwkv\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.968502 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968380 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.968502 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968396 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c5244e9-0e47-4a6d-a27d-24ce8455a537-serving-cert\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.968502 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968422 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.968655 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968557 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:56.968655 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968587 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.968655 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968647 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtb2k\" (UniqueName: \"kubernetes.io/projected/0bbc3b7e-1648-4984-b4da-673a033aca22-kube-api-access-dtb2k\") pod \"volume-data-source-validator-58c9f5f46d-lnvpp\" (UID: \"0bbc3b7e-1648-4984-b4da-673a033aca22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:04:56.968769 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968703 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.968769 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qj2\" (UniqueName: \"kubernetes.io/projected/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-kube-api-access-n4qj2\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.968769 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968760 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.968912 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968781 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-cert\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:56.968912 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968805 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.968912 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968831 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.968912 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.968857 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:04:56.969097 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.968915 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.468896766 +0000 UTC m=+30.652554730 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:04:56.969097 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.968963 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-service-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.969097 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969012 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/745ded03-fed2-450a-bb70-8a78b9d3200d-config\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969095 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969106 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969151 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhs5\" (UniqueName: \"kubernetes.io/projected/82a02683-66a4-43a5-9d32-c53f72d1f613-kube-api-access-vfhs5\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-config\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969217 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.969250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969243 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0964e1df-d8ed-402a-831f-b8996a1236f3-tmp-dir\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969292 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969332 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969370 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969399 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/745ded03-fed2-450a-bb70-8a78b9d3200d-serving-cert\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969439 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jzx\" (UniqueName: \"kubernetes.io/projected/1edd18e2-9aa6-4708-98e9-773c0a38a368-kube-api-access-j6jzx\") pod \"kas-connection-checker-685895c7c9-mvcq4\" (UID: \"1edd18e2-9aa6-4708-98e9-773c0a38a368\") " pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969469 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969500 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:56.969616 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969558 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-service-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.969666 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.469625595 +0000 UTC m=+30.653283569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969794 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969815 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc8tf\" (UniqueName: \"kubernetes.io/projected/0964e1df-d8ed-402a-831f-b8996a1236f3-kube-api-access-jc8tf\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969820 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969830 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-tmp\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969845 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0964e1df-d8ed-402a-831f-b8996a1236f3-config-volume\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969849 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-config\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969911 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd7n4\" (UniqueName: \"kubernetes.io/projected/e8fc0eda-47b5-49fb-86bd-3e23f563c46b-kube-api-access-bd7n4\") pod \"network-check-source-6f78d59f99-dlhnx\" (UID: \"e8fc0eda-47b5-49fb-86bd-3e23f563c46b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:04:56.969945 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969937 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969955 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447n8\" (UniqueName: \"kubernetes.io/projected/7c5244e9-0e47-4a6d-a27d-24ce8455a537-kube-api-access-447n8\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969973 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.969988 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970006 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-snapshots\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970024 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970039 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5vr\" (UniqueName: \"kubernetes.io/projected/31e371fb-f718-43d0-bf0f-d909838ac2d0-kube-api-access-9t5vr\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970134 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmqlg\" (UniqueName: \"kubernetes.io/projected/745ded03-fed2-450a-bb70-8a78b9d3200d-kube-api-access-kmqlg\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae7af53a-1838-409e-b5f3-6aa5c688dd55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970194 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-tmp\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970200 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970244 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.970288 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970242 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsx2q\" (UniqueName: \"kubernetes.io/projected/35ce84ec-29fd-49ab-9c7a-092e99f093b8-kube-api-access-jsx2q\") pod \"kas-connection-checker-685895c7c9-gtptx\" (UID: \"35ce84ec-29fd-49ab-9c7a-092e99f093b8\") " pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970337 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970433 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970466 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-trusted-ca\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970497 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f58b544f-c636-4c93-9e1c-9a817590d307-serving-cert\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970527 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-default-certificate\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970555 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970589 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970773 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:56.970826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970824 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkh5\" (UniqueName: \"kubernetes.io/projected/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-kube-api-access-trkh5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.971191 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.970853 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9fbr\" (UniqueName: \"kubernetes.io/projected/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-kube-api-access-x9fbr\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:56.971191 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.971106 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae7af53a-1838-409e-b5f3-6aa5c688dd55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:56.971191 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.971129 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:56.971191 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.971147 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-snapshots\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.971755 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.971739 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-trusted-ca\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.980757 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.980848 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.980892 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.980902 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.980941 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.981029 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.981030 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.981121 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 00:04:56.981285 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.981126 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-kf6rl\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.982096 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.982163 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert podName:ae7af53a-1838-409e-b5f3-6aa5c688dd55 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.482140009 +0000 UTC m=+30.665797985 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jm2js" (UID: "ae7af53a-1838-409e-b5f3-6aa5c688dd55") : secret "networking-console-plugin-cert" not found Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.982389 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.982888 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.983096 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.983325 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.983586 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.983743 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkhng\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.983941 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984000 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-p464f\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984036 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jkmg8\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984125 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.983746 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984432 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984572 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984712 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984744 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984966 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.984973 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.985216 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.985268 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:04:56.985368 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.985286 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c595ff78b-zkrdt"] Nov 03 00:04:56.986356 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.985484 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rdw2q\"" Nov 03 00:04:56.986356 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.986295 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvvt\" (UniqueName: \"kubernetes.io/projected/d7e0f3c7-a695-4b00-8178-1955b1159469-kube-api-access-thvvt\") pod \"kas-connection-checker-685895c7c9-kzvvs\" (UID: \"d7e0f3c7-a695-4b00-8178-1955b1159469\") " pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:04:56.986460 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.986356 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cd89fb64c-8bxkm"] Nov 03 00:04:56.987484 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.987439 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2pqm"] Nov 03 00:04:56.988130 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.987814 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f58b544f-c636-4c93-9e1c-9a817590d307-serving-cert\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.988685 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.988668 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6"] Nov 03 00:04:56.989940 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.989625 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cdfnj"] Nov 03 00:04:56.989940 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.989808 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:04:56.989940 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.989822 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:04:56.989940 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.989867 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.489851793 +0000 UTC m=+30.673509779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:04:56.990451 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.990357 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:04:56.990451 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.990426 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.490409807 +0000 UTC m=+30.674067766 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:04:56.990995 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.990790 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:04:56.990995 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.990832 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.990995 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:56.990858 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.490843277 +0000 UTC m=+30.674501249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:04:56.990995 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.990944 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp"] Nov 03 00:04:56.991381 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.991362 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs77t"] Nov 03 00:04:56.991645 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.991593 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:56.991845 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.991829 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:56.991994 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.991974 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-stats-auth\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:56.992072 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.992058 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c5244e9-0e47-4a6d-a27d-24ce8455a537-serving-cert\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:56.993250 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.993217 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.993586 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.993571 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:56.995215 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:56.995200 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-default-certificate\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:57.028235 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.027646 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtb2k\" (UniqueName: \"kubernetes.io/projected/0bbc3b7e-1648-4984-b4da-673a033aca22-kube-api-access-dtb2k\") pod \"volume-data-source-validator-58c9f5f46d-lnvpp\" (UID: \"0bbc3b7e-1648-4984-b4da-673a033aca22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:04:57.028235 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.028001 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwkv\" (UniqueName: \"kubernetes.io/projected/f58b544f-c636-4c93-9e1c-9a817590d307-kube-api-access-rpwkv\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:57.028235 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.028195 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5vr\" (UniqueName: \"kubernetes.io/projected/31e371fb-f718-43d0-bf0f-d909838ac2d0-kube-api-access-9t5vr\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:57.028427 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.028346 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:57.028543 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.028502 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jzx\" (UniqueName: \"kubernetes.io/projected/1edd18e2-9aa6-4708-98e9-773c0a38a368-kube-api-access-j6jzx\") pod \"kas-connection-checker-685895c7c9-mvcq4\" (UID: \"1edd18e2-9aa6-4708-98e9-773c0a38a368\") " pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:04:57.029836 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.029794 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447n8\" (UniqueName: \"kubernetes.io/projected/7c5244e9-0e47-4a6d-a27d-24ce8455a537-kube-api-access-447n8\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:57.030058 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.030036 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkh5\" (UniqueName: \"kubernetes.io/projected/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-kube-api-access-trkh5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:57.030144 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.030122 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsx2q\" (UniqueName: \"kubernetes.io/projected/35ce84ec-29fd-49ab-9c7a-092e99f093b8-kube-api-access-jsx2q\") pod \"kas-connection-checker-685895c7c9-gtptx\" (UID: \"35ce84ec-29fd-49ab-9c7a-092e99f093b8\") " pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:04:57.030202 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.030135 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:57.030260 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.030196 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhs5\" (UniqueName: \"kubernetes.io/projected/82a02683-66a4-43a5-9d32-c53f72d1f613-kube-api-access-vfhs5\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:57.030661 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.030621 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qj2\" (UniqueName: \"kubernetes.io/projected/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-kube-api-access-n4qj2\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:57.030981 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.030962 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:57.035772 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.035756 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:04:57.041956 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.041939 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:04:57.047908 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.047892 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:04:57.072001 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.071835 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072123 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmqlg\" (UniqueName: \"kubernetes.io/projected/745ded03-fed2-450a-bb70-8a78b9d3200d-kube-api-access-kmqlg\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.072123 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072102 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072239 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072133 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fbr\" (UniqueName: \"kubernetes.io/projected/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-kube-api-access-x9fbr\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:57.072239 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072168 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072239 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072208 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072369 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072254 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-cert\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:57.072369 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072281 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072369 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072305 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072369 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072336 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/745ded03-fed2-450a-bb70-8a78b9d3200d-config\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.072553 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072383 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0964e1df-d8ed-402a-831f-b8996a1236f3-tmp-dir\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.072553 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072422 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.072553 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072450 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/745ded03-fed2-450a-bb70-8a78b9d3200d-serving-cert\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.072553 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072491 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.072553 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072520 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8tf\" (UniqueName: \"kubernetes.io/projected/0964e1df-d8ed-402a-831f-b8996a1236f3-kube-api-access-jc8tf\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.072553 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072546 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0964e1df-d8ed-402a-831f-b8996a1236f3-config-volume\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.072847 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072571 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7n4\" (UniqueName: \"kubernetes.io/projected/e8fc0eda-47b5-49fb-86bd-3e23f563c46b-kube-api-access-bd7n4\") pod \"network-check-source-6f78d59f99-dlhnx\" (UID: \"e8fc0eda-47b5-49fb-86bd-3e23f563c46b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:04:57.072847 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.072612 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.073487 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.073457 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.073599 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.073581 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.073858 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.073842 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.074658 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.074208 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:04:57.074658 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.074268 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls podName:0964e1df-d8ed-402a-831f-b8996a1236f3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.574250896 +0000 UTC m=+30.757908872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls") pod "dns-default-n2pqm" (UID: "0964e1df-d8ed-402a-831f-b8996a1236f3") : secret "dns-default-metrics-tls" not found Nov 03 00:04:57.074810 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.074714 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/745ded03-fed2-450a-bb70-8a78b9d3200d-config\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.074960 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.074942 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0964e1df-d8ed-402a-831f-b8996a1236f3-tmp-dir\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.076829 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.075787 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0964e1df-d8ed-402a-831f-b8996a1236f3-config-volume\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.076829 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.076485 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:04:57.076829 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.076501 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:04:57.076829 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.076567 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:04:57.576552226 +0000 UTC m=+30.760210196 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:04:57.077473 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.077452 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.077801 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.077777 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/745ded03-fed2-450a-bb70-8a78b9d3200d-serving-cert\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.077878 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.077836 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-cert\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:57.078504 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.078483 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.088043 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.088022 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:04:57.094872 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.094850 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:04:57.100500 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.100482 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:04:57.114912 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.114237 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:04:57.141150 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.133145 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmqlg\" (UniqueName: \"kubernetes.io/projected/745ded03-fed2-450a-bb70-8a78b9d3200d-kube-api-access-kmqlg\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.141150 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.136895 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:04:57.141150 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.139149 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.141150 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.139557 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fbr\" (UniqueName: \"kubernetes.io/projected/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-kube-api-access-x9fbr\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:57.141708 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.141338 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.145000 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.144891 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8tf\" (UniqueName: \"kubernetes.io/projected/0964e1df-d8ed-402a-831f-b8996a1236f3-kube-api-access-jc8tf\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.145959 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.145909 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7n4\" (UniqueName: \"kubernetes.io/projected/e8fc0eda-47b5-49fb-86bd-3e23f563c46b-kube-api-access-bd7n4\") pod \"network-check-source-6f78d59f99-dlhnx\" (UID: \"e8fc0eda-47b5-49fb-86bd-3e23f563c46b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:04:57.150576 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.150403 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:04:57.157846 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.157792 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:04:57.165865 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.165729 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:04:57.218444 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.216812 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:04:57.218444 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.217346 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:04:57.218596 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.218456 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:04:57.273657 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.273596 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:04:57.274186 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.274015 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:04:57.274186 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.274152 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:04:57.278268 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.278247 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:04:57.278372 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.278347 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:04:57.428196 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.427952 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-mvcq4"] Nov 03 00:04:57.431297 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.431240 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-kzvvs"] Nov 03 00:04:57.431415 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.431371 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1edd18e2_9aa6_4708_98e9_773c0a38a368.slice/crio-db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc WatchSource:0}: Error finding container db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc: Status 404 returned error can't find the container with id db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc Nov 03 00:04:57.436672 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.434974 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7e0f3c7_a695_4b00_8178_1955b1159469.slice/crio-094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31 WatchSource:0}: Error finding container 094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31: Status 404 returned error can't find the container with id 094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31 Nov 03 00:04:57.478473 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.477579 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:57.478473 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.477799 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:57.478473 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.477960 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:04:57.478473 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.478010 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.477994369 +0000 UTC m=+31.661652345 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:04:57.478473 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.478078 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.478068355 +0000 UTC m=+31.661726325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:04:57.478933 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.478908 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-xs77t"] Nov 03 00:04:57.483072 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.483047 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8505e4f1_35bd_4ff0_ad33_2febfff8fbf7.slice/crio-6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707 WatchSource:0}: Error finding container 6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707: Status 404 returned error can't find the container with id 6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707 Nov 03 00:04:57.490235 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.490197 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-mdw87"] Nov 03 00:04:57.493287 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.493264 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w"] Nov 03 00:04:57.499876 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.499368 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58b544f_c636_4c93_9e1c_9a817590d307.slice/crio-098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7 WatchSource:0}: Error finding container 098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7: Status 404 returned error can't find the container with id 098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7 Nov 03 00:04:57.499876 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.499568 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33e4c7ae_33c0_4b0d_a1d9_bda9f56a5d59.slice/crio-5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94 WatchSource:0}: Error finding container 5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94: Status 404 returned error can't find the container with id 5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94 Nov 03 00:04:57.532153 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.532129 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-cdfnj"] Nov 03 00:04:57.535961 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.535835 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c5244e9_0e47_4a6d_a27d_24ce8455a537.slice/crio-204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e WatchSource:0}: Error finding container 204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e: Status 404 returned error can't find the container with id 204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e Nov 03 00:04:57.552143 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.552078 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gtptx"] Nov 03 00:04:57.554359 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.554337 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35ce84ec_29fd_49ab_9c7a_092e99f093b8.slice/crio-01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e WatchSource:0}: Error finding container 01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e: Status 404 returned error can't find the container with id 01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e Nov 03 00:04:57.562544 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.562486 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-pruner-29894400-7th8w"] Nov 03 00:04:57.566493 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.566469 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84c8cd4_9ec7_4ae7_bc01_64a0b37f9804.slice/crio-ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262 WatchSource:0}: Error finding container ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262: Status 404 returned error can't find the container with id ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262 Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.578978 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.579021 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.579055 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.579152 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.579193 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.579235 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579355 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579370 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579415 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.579398184 +0000 UTC m=+31.763056152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579480 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579511 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.579501004 +0000 UTC m=+31.763158962 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579567 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579597 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert podName:ae7af53a-1838-409e-b5f3-6aa5c688dd55 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.579588015 +0000 UTC m=+31.763245976 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jm2js" (UID: "ae7af53a-1838-409e-b5f3-6aa5c688dd55") : secret "networking-console-plugin-cert" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579672 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:04:57.579732 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579714 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.57970354 +0000 UTC m=+31.763361498 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:04:57.580577 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579756 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:04:57.580577 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579759 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:04:57.580577 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579775 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:04:57.580577 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579820 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls podName:0964e1df-d8ed-402a-831f-b8996a1236f3 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.579806992 +0000 UTC m=+31.763464948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls") pod "dns-default-n2pqm" (UID: "0964e1df-d8ed-402a-831f-b8996a1236f3") : secret "dns-default-metrics-tls" not found Nov 03 00:04:57.580577 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:57.579834 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:04:58.579828297 +0000 UTC m=+31.763486253 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:04:57.593439 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.593410 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp"] Nov 03 00:04:57.597387 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.597365 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbc3b7e_1648_4984_b4da_673a033aca22.slice/crio-187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03 WatchSource:0}: Error finding container 187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03: Status 404 returned error can't find the container with id 187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03 Nov 03 00:04:57.602259 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.602202 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6"] Nov 03 00:04:57.605353 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.605319 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745ded03_fed2_450a_bb70_8a78b9d3200d.slice/crio-1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18 WatchSource:0}: Error finding container 1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18: Status 404 returned error can't find the container with id 1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18 Nov 03 00:04:57.616169 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:57.616146 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx"] Nov 03 00:04:57.620391 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:04:57.620372 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8fc0eda_47b5_49fb_86bd_3e23f563c46b.slice/crio-c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4 WatchSource:0}: Error finding container c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4: Status 404 returned error can't find the container with id c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4 Nov 03 00:04:58.387091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.387062 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" event={"ID":"e8fc0eda-47b5-49fb-86bd-3e23f563c46b","Type":"ContainerStarted","Data":"c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4"} Nov 03 00:04:58.389329 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.389283 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-7th8w" event={"ID":"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804","Type":"ContainerStarted","Data":"1f47ceb8ed88708d7d92561f3404ddca5575ddf2bd203bea56bbd9f1422243bd"} Nov 03 00:04:58.389458 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.389377 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-7th8w" event={"ID":"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804","Type":"ContainerStarted","Data":"ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262"} Nov 03 00:04:58.394094 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.394059 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" event={"ID":"1edd18e2-9aa6-4708-98e9-773c0a38a368","Type":"ContainerStarted","Data":"d812074f0a3d5fc1e7c7df82b7af0fa42ad1ec63d3315e38a838f807792184a6"} Nov 03 00:04:58.394200 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.394102 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" event={"ID":"1edd18e2-9aa6-4708-98e9-773c0a38a368","Type":"ContainerStarted","Data":"db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc"} Nov 03 00:04:58.395231 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.395211 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" event={"ID":"7c5244e9-0e47-4a6d-a27d-24ce8455a537","Type":"ContainerStarted","Data":"204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e"} Nov 03 00:04:58.396269 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.396228 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" event={"ID":"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59","Type":"ContainerStarted","Data":"5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94"} Nov 03 00:04:58.398484 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.398465 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gtptx" event={"ID":"35ce84ec-29fd-49ab-9c7a-092e99f093b8","Type":"ContainerStarted","Data":"5f98269694b1da0e61c1f594ff273ae111f27e2219b142fd9a6ae6118a83fd48"} Nov 03 00:04:58.398596 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.398501 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-gtptx" event={"ID":"35ce84ec-29fd-49ab-9c7a-092e99f093b8","Type":"ContainerStarted","Data":"01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e"} Nov 03 00:04:58.401728 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.401708 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" event={"ID":"745ded03-fed2-450a-bb70-8a78b9d3200d","Type":"ContainerStarted","Data":"1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18"} Nov 03 00:04:58.402730 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.402712 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mdw87" event={"ID":"f58b544f-c636-4c93-9e1c-9a817590d307","Type":"ContainerStarted","Data":"098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7"} Nov 03 00:04:58.404021 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.404001 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" event={"ID":"d7e0f3c7-a695-4b00-8178-1955b1159469","Type":"ContainerStarted","Data":"796f6460de57565ad1d3bec335611a894776cb0e434e1c53b275a7165270eb51"} Nov 03 00:04:58.404091 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.404039 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" event={"ID":"d7e0f3c7-a695-4b00-8178-1955b1159469","Type":"ContainerStarted","Data":"094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31"} Nov 03 00:04:58.405020 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.404998 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" event={"ID":"0bbc3b7e-1648-4984-b4da-673a033aca22","Type":"ContainerStarted","Data":"187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03"} Nov 03 00:04:58.405983 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.405967 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs77t" event={"ID":"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7","Type":"ContainerStarted","Data":"6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707"} Nov 03 00:04:58.486377 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.486315 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:04:58.486504 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.486414 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:58.486563 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.486501 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:04:58.486612 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.486565 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.486546321 +0000 UTC m=+33.670204308 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:04:58.486612 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.486584 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.486571295 +0000 UTC m=+33.670229280 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:04:58.587439 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.587404 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:04:58.587589 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.587464 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:04:58.587589 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.587506 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:04:58.587589 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587547 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:04:58.587738 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.587611 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:04:58.587738 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587624 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.587606109 +0000 UTC m=+33.771264086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:04:58.587738 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587623 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:04:58.587738 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587696 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:04:58.587738 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587708 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:04:58.587738 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587721 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert podName:ae7af53a-1838-409e-b5f3-6aa5c688dd55 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.587702186 +0000 UTC m=+33.771360166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jm2js" (UID: "ae7af53a-1838-409e-b5f3-6aa5c688dd55") : secret "networking-console-plugin-cert" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587769 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.587755356 +0000 UTC m=+33.771413332 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.587762 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587800 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587824 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:58.587825 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587851 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.58782277 +0000 UTC m=+33.771480727 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587869 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls podName:0964e1df-d8ed-402a-831f-b8996a1236f3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.587861224 +0000 UTC m=+33.771519180 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls") pod "dns-default-n2pqm" (UID: "0964e1df-d8ed-402a-831f-b8996a1236f3") : secret "dns-default-metrics-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587881 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587889 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:04:58.588034 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:04:58.587910 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.587904064 +0000 UTC m=+33.771562046 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:04:59.253481 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:59.253422 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-pruner-29894400-7th8w" podStartSLOduration=299.253403327 podStartE2EDuration="4m59.253403327s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:59.085274957 +0000 UTC m=+32.268932938" watchObservedRunningTime="2026-11-03 00:04:59.253403327 +0000 UTC m=+32.437061308" Nov 03 00:04:59.368417 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:59.368362 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" podStartSLOduration=441.368343478 podStartE2EDuration="7m21.368343478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:59.254725241 +0000 UTC m=+32.438383220" watchObservedRunningTime="2026-11-03 00:04:59.368343478 +0000 UTC m=+32.552001459" Nov 03 00:04:59.369595 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:59.369546 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-gtptx" podStartSLOduration=441.369535407 podStartE2EDuration="7m21.369535407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:59.369205789 +0000 UTC m=+32.552863767" watchObservedRunningTime="2026-11-03 00:04:59.369535407 +0000 UTC m=+32.553193383" Nov 03 00:04:59.483570 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:04:59.483118 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" podStartSLOduration=441.483101704 podStartE2EDuration="7m21.483101704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:59.481358485 +0000 UTC m=+32.665016465" watchObservedRunningTime="2026-11-03 00:04:59.483101704 +0000 UTC m=+32.666759683" Nov 03 00:05:00.507072 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.507041 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:00.507698 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.507671 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:00.519673 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.515809 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:00.519673 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.515875 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.515856828 +0000 UTC m=+37.699514802 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:00.519673 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.516086 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.516065554 +0000 UTC m=+37.699723524 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:05:00.609501 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.609463 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.609529 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.609570 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.609608 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.609652 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609658 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:00.609685 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:00.609713 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609693 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609762 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.609739757 +0000 UTC m=+37.793397737 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609813 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609837 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609848 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609867 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.609850463 +0000 UTC m=+37.793508422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609886 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.609873121 +0000 UTC m=+37.793531100 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609936 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609950 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609972 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.609960875 +0000 UTC m=+37.793618851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.609988 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert podName:ae7af53a-1838-409e-b5f3-6aa5c688dd55 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.60997952 +0000 UTC m=+37.793637482 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jm2js" (UID: "ae7af53a-1838-409e-b5f3-6aa5c688dd55") : secret "networking-console-plugin-cert" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.610014 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:05:00.610080 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:00.610044 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls podName:0964e1df-d8ed-402a-831f-b8996a1236f3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.610033519 +0000 UTC m=+37.793691480 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls") pod "dns-default-n2pqm" (UID: "0964e1df-d8ed-402a-831f-b8996a1236f3") : secret "dns-default-metrics-tls" not found Nov 03 00:05:01.016077 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.015035 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:01.016077 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.015239 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:01.016077 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:01.015381 2573 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:05:01.016077 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:01.015444 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:05:33.0154256 +0000 UTC m=+66.199083561 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : secret "metrics-daemon-secret" not found Nov 03 00:05:01.027134 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.027025 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:01.117287 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.116803 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:01.121427 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.121369 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:01.137078 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.137046 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:01.164676 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.164565 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:01.419966 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.419876 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" event={"ID":"0bbc3b7e-1648-4984-b4da-673a033aca22","Type":"ContainerStarted","Data":"453dc2ef95169936011bbe789eb1f47b50dc24a4bdf7aa8ca2e50e966574e3de"} Nov 03 00:05:01.535098 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.534812 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" podStartSLOduration=412.690720163 podStartE2EDuration="6m55.534794584s" totalImagesPullingTime="2.844074421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:01.533189487 +0000 UTC m=+34.716847466" watchObservedRunningTime="2026-11-03 00:05:01.534794584 +0000 UTC m=+34.718452564" Nov 03 00:05:01.588216 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.588131 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cw8kk"] Nov 03 00:05:01.592979 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:05:01.592942 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee7c64c_4945_469a_8d27_cfb702aabf50.slice/crio-4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e WatchSource:0}: Error finding container 4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e: Status 404 returned error can't find the container with id 4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e Nov 03 00:05:01.626622 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:01.626589 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gbt9l"] Nov 03 00:05:01.659641 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:05:01.659605 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f87535b_ae3b_45e3_abd6_ca3ca4b9bdb1.slice/crio-55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82 WatchSource:0}: Error finding container 55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82: Status 404 returned error can't find the container with id 55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82 Nov 03 00:05:02.425458 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.425419 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/0.log" Nov 03 00:05:02.425617 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.425471 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c5244e9-0e47-4a6d-a27d-24ce8455a537" containerID="0dd514427897d6d190dec6d0caa9c9254778a426b4d75588731a7a1bbb6b955e" exitCode=255 Nov 03 00:05:02.425617 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.425571 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" event={"ID":"7c5244e9-0e47-4a6d-a27d-24ce8455a537","Type":"ContainerDied","Data":"0dd514427897d6d190dec6d0caa9c9254778a426b4d75588731a7a1bbb6b955e"} Nov 03 00:05:02.425813 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.425779 2573 scope.go:122] "RemoveContainer" containerID="0dd514427897d6d190dec6d0caa9c9254778a426b4d75588731a7a1bbb6b955e" Nov 03 00:05:02.430480 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.427487 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mdw87" event={"ID":"f58b544f-c636-4c93-9e1c-9a817590d307","Type":"ContainerStarted","Data":"11cbea1dfb216d6eef0305df8d92fb9e5ab9cf85c29a209fc0a9be03aaba9fa2"} Nov 03 00:05:02.430480 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.428760 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbt9l" event={"ID":"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1","Type":"ContainerStarted","Data":"55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82"} Nov 03 00:05:02.430480 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:02.430118 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cw8kk" event={"ID":"2ee7c64c-4945-469a-8d27-cfb702aabf50","Type":"ContainerStarted","Data":"4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e"} Nov 03 00:05:03.436468 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.436436 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:05:03.436948 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.436906 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/0.log" Nov 03 00:05:03.436996 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.436951 2573 generic.go:356] "Generic (PLEG): container finished" podID="7c5244e9-0e47-4a6d-a27d-24ce8455a537" containerID="0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435" exitCode=255 Nov 03 00:05:03.437105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.437028 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" event={"ID":"7c5244e9-0e47-4a6d-a27d-24ce8455a537","Type":"ContainerDied","Data":"0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435"} Nov 03 00:05:03.437105 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.437096 2573 scope.go:122] "RemoveContainer" containerID="0dd514427897d6d190dec6d0caa9c9254778a426b4d75588731a7a1bbb6b955e" Nov 03 00:05:03.437281 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.437265 2573 scope.go:122] "RemoveContainer" containerID="0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435" Nov 03 00:05:03.437505 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:03.437481 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cdfnj_openshift-console-operator(7c5244e9-0e47-4a6d-a27d-24ce8455a537)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" podUID="7c5244e9-0e47-4a6d-a27d-24ce8455a537" Nov 03 00:05:03.440129 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.440056 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" event={"ID":"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59","Type":"ContainerStarted","Data":"a97ecefc68afeeaeb689cdf2c4fd077d1da3d290e9460340d0ab6cca7cd1426d"} Nov 03 00:05:03.453764 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.453695 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-xs77t" event={"ID":"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7","Type":"ContainerStarted","Data":"52aa814a9b4695d195ba7965ea72623b82e0a59197d6a2991690a75a42bc4e07"} Nov 03 00:05:03.560866 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.557320 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-mdw87" podStartSLOduration=403.941825632 podStartE2EDuration="6m48.557282541s" totalImagesPullingTime="4.615456909s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:02.655271868 +0000 UTC m=+35.838929847" watchObservedRunningTime="2026-11-03 00:05:03.557282541 +0000 UTC m=+36.740940520" Nov 03 00:05:03.560866 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.559502 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-llwll"] Nov 03 00:05:03.590052 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.590016 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.643825 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.643570 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:05:03.643825 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.643667 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:05:03.643825 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.643797 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5tjj\"" Nov 03 00:05:03.663692 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.663666 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llwll"] Nov 03 00:05:03.668506 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.668456 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" podStartSLOduration=411.367982967 podStartE2EDuration="6m56.668442441s" totalImagesPullingTime="5.300459474s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:03.666276697 +0000 UTC m=+36.849934675" watchObservedRunningTime="2026-11-03 00:05:03.668442441 +0000 UTC m=+36.852100419" Nov 03 00:05:03.756193 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.756160 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-data-volume\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.756334 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.756222 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.756439 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.756411 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.756500 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.756470 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-crio-socket\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.756680 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.756661 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wr9s\" (UniqueName: \"kubernetes.io/projected/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-api-access-9wr9s\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.767544 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.767516 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-rj2h8-pnnjn"] Nov 03 00:05:03.778890 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.778847 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-xs77t" podStartSLOduration=2.715456962 podStartE2EDuration="7.778831751s" totalImagesPullingTime="5.063374789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:03.777683526 +0000 UTC m=+36.961341502" watchObservedRunningTime="2026-11-03 00:05:03.778831751 +0000 UTC m=+36.962489731" Nov 03 00:05:03.790583 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.790035 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:03.827218 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.827188 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rj2h8-pnnjn"] Nov 03 00:05:03.842174 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.842132 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-klhlc\"" Nov 03 00:05:03.858113 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.858075 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wr9s\" (UniqueName: \"kubernetes.io/projected/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-api-access-9wr9s\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.858262 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.858134 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-data-volume\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.858262 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.858157 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.858380 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.858309 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.858380 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.858354 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-crio-socket\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.858492 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:03.858472 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:05:03.858548 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.858526 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-crio-socket\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.858548 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:03.858543 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.35852066 +0000 UTC m=+37.542178633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : configmap "kube-rbac-proxy" not found Nov 03 00:05:03.858677 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:03.858617 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:03.858794 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:03.858685 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:04.358669921 +0000 UTC m=+37.542327898 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:03.859197 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.859160 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-data-volume\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.923780 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.915931 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wr9s\" (UniqueName: \"kubernetes.io/projected/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-api-access-9wr9s\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:03.959193 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.959164 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:03.959334 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.959212 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:03.959403 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.959378 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:03.959670 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:03.959506 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.060240 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.060165 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.060240 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.060230 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.061993 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.061961 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.062129 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.062103 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.062661 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.062619 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.062771 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.062755 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.062934 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.062915 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.118455 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.118425 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.121230 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.121203 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:04.365249 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.365177 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:04.365401 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.365274 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:04.365401 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.365312 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:05:04.365509 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.365414 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:04.365509 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.365423 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:05.365379398 +0000 UTC m=+38.549037368 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : configmap "kube-rbac-proxy" not found Nov 03 00:05:04.365509 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.365494 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:05.365475414 +0000 UTC m=+38.549133376 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:04.460826 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.460795 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:05:04.461816 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.461709 2573 scope.go:122] "RemoveContainer" containerID="0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435" Nov 03 00:05:04.462519 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.461891 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cdfnj_openshift-console-operator(7c5244e9-0e47-4a6d-a27d-24ce8455a537)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" podUID="7c5244e9-0e47-4a6d-a27d-24ce8455a537" Nov 03 00:05:04.568573 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.568541 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:04.568740 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.568719 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:04.571193 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.569809 2573 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:04.571193 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.569864 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.569850748 +0000 UTC m=+45.753508705 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:04.571193 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.570298 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.570279056 +0000 UTC m=+45.753937012 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:05:04.670063 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.670026 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:04.670196 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.670098 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:04.670196 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.670136 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:04.670312 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.670194 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:04.670312 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.670224 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:04.670312 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.670260 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:04.670494 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670478 2573 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:05:04.670550 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670540 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.670522371 +0000 UTC m=+45.854180330 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:05:04.670625 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670614 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:04.670625 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670646 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:05:04.670746 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670680 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.670669777 +0000 UTC m=+45.854327750 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:05:04.670746 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670741 2573 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 00:05:04.670836 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670768 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls podName:0964e1df-d8ed-402a-831f-b8996a1236f3 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.670758685 +0000 UTC m=+45.854416644 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls") pod "dns-default-n2pqm" (UID: "0964e1df-d8ed-402a-831f-b8996a1236f3") : secret "dns-default-metrics-tls" not found Nov 03 00:05:04.670836 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670823 2573 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:04.670836 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670832 2573 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:05:04.670967 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670858 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.670849577 +0000 UTC m=+45.854507535 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:05:04.670967 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670910 2573 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:05:04.670967 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670937 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.670928146 +0000 UTC m=+45.854586104 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:05:04.671103 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.670994 2573 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 00:05:04.671103 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:04.671024 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert podName:ae7af53a-1838-409e-b5f3-6aa5c688dd55 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.671014301 +0000 UTC m=+45.854672260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-jm2js" (UID: "ae7af53a-1838-409e-b5f3-6aa5c688dd55") : secret "networking-console-plugin-cert" not found Nov 03 00:05:04.882455 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:04.882154 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-rj2h8-pnnjn"] Nov 03 00:05:04.887287 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:05:04.887253 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9425f6e8_123b_4acb_a3d1_5ccd19761401.slice/crio-26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f WatchSource:0}: Error finding container 26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f: Status 404 returned error can't find the container with id 26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f Nov 03 00:05:05.026137 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.025876 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd"] Nov 03 00:05:05.048455 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.048422 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4"] Nov 03 00:05:05.048573 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.048538 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:05.065100 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.064833 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:05.083686 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.083662 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd"] Nov 03 00:05:05.093292 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.093249 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4"] Nov 03 00:05:05.106031 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.105982 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 00:05:05.106140 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.106044 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:05.106309 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.106244 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-52krg\"" Nov 03 00:05:05.175509 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.175447 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7w8d\" (UniqueName: \"kubernetes.io/projected/ed231fcb-17c8-4887-b2b3-1fb1c64eb442-kube-api-access-b7w8d\") pod \"migrator-66df5d496b-zqsm4\" (UID: \"ed231fcb-17c8-4887-b2b3-1fb1c64eb442\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:05.175509 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.175513 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb28b\" (UniqueName: \"kubernetes.io/projected/66a8deff-4567-470f-98d0-bec9ea590d9a-kube-api-access-hb28b\") pod \"migrator-66df5d496b-xbbfd\" (UID: \"66a8deff-4567-470f-98d0-bec9ea590d9a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:05.276363 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.276327 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7w8d\" (UniqueName: \"kubernetes.io/projected/ed231fcb-17c8-4887-b2b3-1fb1c64eb442-kube-api-access-b7w8d\") pod \"migrator-66df5d496b-zqsm4\" (UID: \"ed231fcb-17c8-4887-b2b3-1fb1c64eb442\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:05.276508 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.276393 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb28b\" (UniqueName: \"kubernetes.io/projected/66a8deff-4567-470f-98d0-bec9ea590d9a-kube-api-access-hb28b\") pod \"migrator-66df5d496b-xbbfd\" (UID: \"66a8deff-4567-470f-98d0-bec9ea590d9a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:05.333710 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.333680 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb28b\" (UniqueName: \"kubernetes.io/projected/66a8deff-4567-470f-98d0-bec9ea590d9a-kube-api-access-hb28b\") pod \"migrator-66df5d496b-xbbfd\" (UID: \"66a8deff-4567-470f-98d0-bec9ea590d9a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:05.333928 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.333909 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7w8d\" (UniqueName: \"kubernetes.io/projected/ed231fcb-17c8-4887-b2b3-1fb1c64eb442-kube-api-access-b7w8d\") pod \"migrator-66df5d496b-zqsm4\" (UID: \"ed231fcb-17c8-4887-b2b3-1fb1c64eb442\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:05.377017 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.376991 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:05.377136 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.377082 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:05.377251 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.377236 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:05.380653 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:05.377601 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:05:05.380653 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:05.377685 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.377667236 +0000 UTC m=+40.561325211 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : configmap "kube-rbac-proxy" not found Nov 03 00:05:05.380653 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:05.378782 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:05.380653 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:05.378842 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:07.378826777 +0000 UTC m=+40.562484750 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:05.399672 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.399626 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:05.470648 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.470602 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" event={"ID":"745ded03-fed2-450a-bb70-8a78b9d3200d","Type":"ContainerStarted","Data":"e59698448079a90b47e802ecc947a2d107959ef018fe1401fd7398a57ce44a4e"} Nov 03 00:05:05.472210 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.472180 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gbt9l" event={"ID":"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1","Type":"ContainerStarted","Data":"baa79def44ee0682081af66b5f8ed9c66070f7428f6d8d7d05ac941c657a84da"} Nov 03 00:05:05.472724 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.472623 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:05.474354 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.474301 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" event={"ID":"9425f6e8-123b-4acb-a3d1-5ccd19761401","Type":"ContainerStarted","Data":"ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167"} Nov 03 00:05:05.474875 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.474585 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" event={"ID":"9425f6e8-123b-4acb-a3d1-5ccd19761401","Type":"ContainerStarted","Data":"26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f"} Nov 03 00:05:05.477274 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.476961 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" event={"ID":"e8fc0eda-47b5-49fb-86bd-3e23f563c46b","Type":"ContainerStarted","Data":"44998e2b03bbce8e40a262f53051dc07e281a4c28a72ef658656e03866d0a26d"} Nov 03 00:05:05.587182 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.587135 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" podStartSLOduration=412.673953112 podStartE2EDuration="6m59.587122649s" totalImagesPullingTime="6.913169537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:05.585826184 +0000 UTC m=+38.769484164" watchObservedRunningTime="2026-11-03 00:05:05.587122649 +0000 UTC m=+38.770780655" Nov 03 00:05:05.698914 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.698813 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" podStartSLOduration=2.698796435 podStartE2EDuration="2.698796435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:05.698119715 +0000 UTC m=+38.881777692" watchObservedRunningTime="2026-11-03 00:05:05.698796435 +0000 UTC m=+38.882454413" Nov 03 00:05:05.771825 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.771789 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd"] Nov 03 00:05:05.774921 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:05:05.774897 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66a8deff_4567_470f_98d0_bec9ea590d9a.slice/crio-534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e WatchSource:0}: Error finding container 534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e: Status 404 returned error can't find the container with id 534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e Nov 03 00:05:05.810718 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.810672 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" podStartSLOduration=402.923894913 podStartE2EDuration="6m49.810657314s" totalImagesPullingTime="6.886762401s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 23:58:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:05.809874617 +0000 UTC m=+38.993532621" watchObservedRunningTime="2026-11-03 00:05:05.810657314 +0000 UTC m=+38.994315288" Nov 03 00:05:05.821808 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.821784 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4"] Nov 03 00:05:05.824624 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:05:05.824585 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded231fcb_17c8_4887_b2b3_1fb1c64eb442.slice/crio-6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce WatchSource:0}: Error finding container 6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce: Status 404 returned error can't find the container with id 6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce Nov 03 00:05:05.924173 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:05.924113 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gbt9l" podStartSLOduration=34.246610865 podStartE2EDuration="37.924092699s" totalImagesPullingTime="3.677481834s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:05.922313239 +0000 UTC m=+39.105971217" watchObservedRunningTime="2026-11-03 00:05:05.924092699 +0000 UTC m=+39.107750678" Nov 03 00:05:06.481736 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:06.481699 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" event={"ID":"ed231fcb-17c8-4887-b2b3-1fb1c64eb442","Type":"ContainerStarted","Data":"6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce"} Nov 03 00:05:06.483188 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:06.483159 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" event={"ID":"66a8deff-4567-470f-98d0-bec9ea590d9a","Type":"ContainerStarted","Data":"534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e"} Nov 03 00:05:07.101216 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.101132 2573 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:07.101365 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.101255 2573 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:07.102004 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.101717 2573 scope.go:122] "RemoveContainer" containerID="0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435" Nov 03 00:05:07.102004 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:07.101955 2573 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-cdfnj_openshift-console-operator(7c5244e9-0e47-4a6d-a27d-24ce8455a537)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" podUID="7c5244e9-0e47-4a6d-a27d-24ce8455a537" Nov 03 00:05:07.132371 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.132342 2573 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sc8zr"] Nov 03 00:05:07.156513 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.156483 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.196381 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.196353 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sc8zr"] Nov 03 00:05:07.244338 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.244129 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 00:05:07.244338 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.244305 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 00:05:07.258040 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.258000 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 00:05:07.258170 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.258145 2573 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gj4lf\"" Nov 03 00:05:07.258238 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.258219 2573 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 00:05:07.304232 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.304197 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgq4q\" (UniqueName: \"kubernetes.io/projected/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-kube-api-access-dgq4q\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.304393 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.304258 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-key\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.304393 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.304284 2573 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-cabundle\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.405813 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.405736 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq4q\" (UniqueName: \"kubernetes.io/projected/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-kube-api-access-dgq4q\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.405813 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.405795 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-key\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.406018 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.405821 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-cabundle\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.406018 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.405865 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:07.406018 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.405941 2573 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:07.406171 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:07.406100 2573 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:07.406171 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:07.406158 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.40614106 +0000 UTC m=+44.589799036 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:07.406338 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:07.406294 2573 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 00:05:07.406419 ip-10-0-143-162 kubenswrapper[2573]: E1103 00:05:07.406351 2573 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.406335665 +0000 UTC m=+44.589993635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : configmap "kube-rbac-proxy" not found Nov 03 00:05:07.406736 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.406679 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-cabundle\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.409181 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.409161 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-key\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.462344 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.462314 2573 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq4q\" (UniqueName: \"kubernetes.io/projected/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-kube-api-access-dgq4q\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.479108 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.479080 2573 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:07.891026 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.886112 2573 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-sc8zr"] Nov 03 00:05:07.891026 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:07.886682 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vplk_88200eac-4b64-4019-b733-335740bfe1e5/dns-node-resolver/0.log" Nov 03 00:05:07.891026 ip-10-0-143-162 kubenswrapper[2573]: W1103 00:05:07.888911 2573 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e4053_bef9_482f_8fe1_44a0d9bd4dfd.slice/crio-846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8 WatchSource:0}: Error finding container 846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8: Status 404 returned error can't find the container with id 846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8 Nov 03 00:05:08.493328 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:08.493296 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" event={"ID":"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd","Type":"ContainerStarted","Data":"22a1384f9645d07c8f49fd6313b955a897b8d604832c7367b3108f47bcb9fca9"} Nov 03 00:05:08.493497 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:08.493371 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" event={"ID":"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd","Type":"ContainerStarted","Data":"846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8"} Nov 03 00:05:08.551440 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:08.551390 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29894400-7th8w_a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804/image-pruner/0.log" Nov 03 00:05:08.609829 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:08.609764 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" podStartSLOduration=1.609741954 podStartE2EDuration="1.609741954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:08.607276418 +0000 UTC m=+41.790934413" watchObservedRunningTime="2026-11-03 00:05:08.609741954 +0000 UTC m=+41.793399934" Nov 03 00:05:09.080762 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:09.080734 2573 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dgbsh_f0f94dde-4d2f-4969-b7ce-357282dc6529/node-ca/0.log" Nov 03 00:05:09.501687 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:09.501650 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" event={"ID":"ed231fcb-17c8-4887-b2b3-1fb1c64eb442","Type":"ContainerStarted","Data":"6cdc25bbd20e8f41208570567818c85a65fee7bfed4f489942472c57261d9cc7"} Nov 03 00:05:09.504989 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:09.504962 2573 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" event={"ID":"66a8deff-4567-470f-98d0-bec9ea590d9a","Type":"ContainerStarted","Data":"73b227b4ccdc518b3b6bcf07878ad04a6d74ebf46a7db0a1614c53998317a20d"} Nov 03 00:05:09.615743 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:09.615680 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" podStartSLOduration=2.746084996 podStartE2EDuration="5.615662002s" totalImagesPullingTime="2.869577006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:09.61511136 +0000 UTC m=+42.798769362" watchObservedRunningTime="2026-11-03 00:05:09.615662002 +0000 UTC m=+42.799319975" Nov 03 00:05:09.728670 ip-10-0-143-162 kubenswrapper[2573]: I1103 00:05:09.728358 2573 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" podStartSLOduration=2.8065305240000002 podStartE2EDuration="5.728338683s" totalImagesPullingTime="2.921808159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:09.727923035 +0000 UTC m=+42.911581010" watchObservedRunningTime="2026-11-03 00:05:09.728338683 +0000 UTC m=+42.911996660" Nov 03 00:05:09.775343 ip-10-0-143-162 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 00:05:09.788148 ip-10-0-143-162 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 00:05:09.788312 ip-10-0-143-162 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 00:05:09.788484 ip-10-0-143-162 systemd[1]: kubelet.service: Consumed 3.135s CPU time. Nov 03 00:05:09.804253 ip-10-0-143-162 systemd[1]: Starting Kubernetes Kubelet... Nov 03 00:05:09.915236 ip-10-0-143-162 kubenswrapper[5144]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:09.915236 ip-10-0-143-162 kubenswrapper[5144]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 00:05:09.915236 ip-10-0-143-162 kubenswrapper[5144]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:09.915236 ip-10-0-143-162 kubenswrapper[5144]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 00:05:09.917460 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917372 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.917460 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917459 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.917460 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917463 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917466 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917469 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917472 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917474 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917477 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917479 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917482 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917484 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917488 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917490 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917492 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917495 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917498 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917501 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917503 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917506 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917508 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917512 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.917572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917514 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917517 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917520 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917522 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917525 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917527 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917529 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917532 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917534 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917537 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917539 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917542 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917544 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917548 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917550 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917553 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917555 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917557 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917561 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.918074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917564 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917568 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917572 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917575 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917578 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917582 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917586 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917590 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917593 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917595 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917598 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917601 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917604 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917607 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917610 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917613 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917615 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917618 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917621 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.918562 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917624 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917646 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917650 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917652 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917655 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917658 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917661 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917663 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917666 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917668 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917671 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917674 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917676 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917679 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917682 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917684 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917686 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917689 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917691 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917693 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.919027 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917696 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917699 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917701 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917704 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917707 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917710 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917712 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917716 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917718 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917721 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917723 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917726 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917728 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917730 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917733 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917736 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917738 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917742 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917744 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.919494 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917747 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917749 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917751 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917754 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917756 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917759 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917761 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.917764 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918537 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918545 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918549 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918552 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918554 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918558 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918560 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918563 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918566 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918568 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918571 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.919995 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918573 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918576 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918578 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918581 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918583 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918586 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918588 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918591 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918594 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918596 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918599 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918602 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918605 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918607 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918610 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918612 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918615 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918617 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918620 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.920432 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918624 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918677 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918683 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918686 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918689 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918691 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918694 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918696 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918699 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918701 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918704 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918706 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918709 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918712 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918714 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918718 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918721 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918723 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918726 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.920906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918728 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918731 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918733 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918736 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918739 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918741 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918743 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918746 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918748 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918751 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918753 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918756 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918758 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918760 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918763 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918765 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918777 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918781 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918784 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.921350 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918786 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918789 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918791 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918794 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918796 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918798 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918801 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918803 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918806 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918809 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918812 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918814 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918816 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918819 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918823 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918825 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918828 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918830 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918833 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918835 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.921802 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918837 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918840 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918842 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918844 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918847 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918849 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918851 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918854 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918857 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918859 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918867 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918870 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918872 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918875 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918877 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918879 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918882 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.918884 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919360 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919365 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.922336 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919369 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919371 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919374 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919377 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919379 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919382 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919384 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919387 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919389 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919392 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919394 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919397 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919399 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919402 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919404 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919407 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919409 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919412 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919416 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919420 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.922829 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919423 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919438 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919448 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919451 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919453 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919456 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919459 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919461 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919464 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919467 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919469 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919471 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919474 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919476 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919479 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919481 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919483 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919486 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919489 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919491 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.923389 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919494 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919496 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919499 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919501 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919505 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919507 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919510 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919512 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919515 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919517 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919520 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919522 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919524 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919527 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919529 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919537 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919540 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919542 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919544 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.923972 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919547 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919550 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919552 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919555 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919557 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919560 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919564 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919568 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919571 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919575 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919579 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919582 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919584 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919587 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919590 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919594 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919598 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919602 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919607 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919610 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.924428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919614 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919617 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919621 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919625 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919644 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919648 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919652 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919656 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919659 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919671 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919674 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919676 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919679 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919681 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919684 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919686 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919689 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919693 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919695 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919698 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.924906 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919700 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919703 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919705 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919708 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.919710 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919808 5144 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919834 5144 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919840 5144 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919845 5144 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919849 5144 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919852 5144 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919857 5144 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919861 5144 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919864 5144 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919867 5144 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919870 5144 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919873 5144 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919876 5144 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919879 5144 flags.go:64] FLAG: --cgroup-root="" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919882 5144 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919885 5144 flags.go:64] FLAG: --client-ca-file="" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919888 5144 flags.go:64] FLAG: --cloud-provider="external" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919891 5144 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 00:05:09.925410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919895 5144 flags.go:64] FLAG: --cluster-domain="" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919898 5144 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919901 5144 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919904 5144 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919907 5144 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919911 5144 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919913 5144 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919916 5144 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919920 5144 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919922 5144 flags.go:64] FLAG: --contention-profiling="false" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919925 5144 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919928 5144 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919931 5144 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919934 5144 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919938 5144 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919940 5144 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919943 5144 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919946 5144 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919949 5144 flags.go:64] FLAG: --enable-server="true" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919951 5144 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919958 5144 flags.go:64] FLAG: --event-burst="100" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919961 5144 flags.go:64] FLAG: --event-qps="50" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919963 5144 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919966 5144 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 00:05:09.925969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919969 5144 flags.go:64] FLAG: --eviction-hard="" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919974 5144 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919976 5144 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919979 5144 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919982 5144 flags.go:64] FLAG: --eviction-soft="" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919985 5144 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919988 5144 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.919994 5144 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920002 5144 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920005 5144 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920008 5144 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920011 5144 flags.go:64] FLAG: --feature-gates="" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920015 5144 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920018 5144 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920021 5144 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920024 5144 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920026 5144 flags.go:64] FLAG: --healthz-port="10248" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920029 5144 flags.go:64] FLAG: --help="false" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920032 5144 flags.go:64] FLAG: --hostname-override="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920035 5144 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920038 5144 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920041 5144 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920044 5144 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 00:05:09.926780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920047 5144 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920050 5144 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920053 5144 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920055 5144 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920058 5144 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920061 5144 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920064 5144 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920067 5144 flags.go:64] FLAG: --kube-reserved="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920069 5144 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920072 5144 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920076 5144 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920079 5144 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920082 5144 flags.go:64] FLAG: --lock-file="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920085 5144 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920087 5144 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920090 5144 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920097 5144 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920100 5144 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920108 5144 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920111 5144 flags.go:64] FLAG: --logging-format="text" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920114 5144 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920117 5144 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920120 5144 flags.go:64] FLAG: --manifest-url="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920123 5144 flags.go:64] FLAG: --manifest-url-header="" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920130 5144 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 00:05:09.927566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920133 5144 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920141 5144 flags.go:64] FLAG: --max-pods="110" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920143 5144 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920146 5144 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920149 5144 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920152 5144 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920155 5144 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920157 5144 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920160 5144 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920167 5144 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920169 5144 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920173 5144 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920175 5144 flags.go:64] FLAG: --pod-cidr="" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920178 5144 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920181 5144 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920184 5144 flags.go:64] FLAG: --pods-per-core="0" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920187 5144 flags.go:64] FLAG: --port="10250" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920191 5144 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920193 5144 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-0865646be73cbdacd" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920196 5144 flags.go:64] FLAG: --qos-reserved="" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920199 5144 flags.go:64] FLAG: --read-only-port="10255" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920202 5144 flags.go:64] FLAG: --register-node="true" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920205 5144 flags.go:64] FLAG: --register-with-taints="" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920209 5144 flags.go:64] FLAG: --registry-burst="10" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920212 5144 flags.go:64] FLAG: --registry-qps="5" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920215 5144 flags.go:64] FLAG: --reserved-cpus="" Nov 03 00:05:09.928210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920223 5144 flags.go:64] FLAG: --reserved-memory="" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920227 5144 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920230 5144 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920232 5144 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920235 5144 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920238 5144 flags.go:64] FLAG: --runonce="false" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920241 5144 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920244 5144 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920247 5144 flags.go:64] FLAG: --seccomp-default="false" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920249 5144 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920252 5144 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920255 5144 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920258 5144 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920261 5144 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920263 5144 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920266 5144 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920269 5144 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920272 5144 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920274 5144 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920277 5144 flags.go:64] FLAG: --system-cgroups="" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920280 5144 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920285 5144 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920287 5144 flags.go:64] FLAG: --tls-cert-file="" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920291 5144 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920298 5144 flags.go:64] FLAG: --tls-min-version="" Nov 03 00:05:09.928853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920300 5144 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920303 5144 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920306 5144 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920308 5144 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920313 5144 flags.go:64] FLAG: --v="2" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920317 5144 flags.go:64] FLAG: --version="false" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920321 5144 flags.go:64] FLAG: --vmodule="" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920325 5144 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920330 5144 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920460 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920464 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920466 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920469 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920471 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920474 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920477 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920480 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920482 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920485 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920487 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920490 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920492 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.929455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920495 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920497 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920500 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920502 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920505 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920507 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920509 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920512 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920516 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920518 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920520 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920523 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920525 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920527 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920531 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920533 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920536 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920538 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920541 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.929977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920544 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920546 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920549 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920551 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920553 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920556 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920559 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920561 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920564 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920566 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920568 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920571 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920574 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920576 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920579 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920581 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920583 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920586 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920588 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920591 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.930530 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920593 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920597 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920599 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920602 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920606 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920609 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920612 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920616 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920619 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920622 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920625 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920641 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920644 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920647 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920650 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920653 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920655 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920658 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920660 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.931104 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920663 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920666 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920669 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920671 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920674 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920676 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920679 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920681 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920683 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920686 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920688 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920691 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920693 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920695 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920699 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920701 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920704 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920706 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920709 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.931737 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920711 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920714 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920717 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920719 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920722 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920724 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920727 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920729 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920733 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920737 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920740 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920744 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920750 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920753 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920755 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920758 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.932385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.920783 5144 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920818 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920823 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920826 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920829 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920831 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920834 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920836 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920839 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920841 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920844 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920847 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920850 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920852 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920856 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920860 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920863 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920866 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920869 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.932905 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920872 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920875 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920877 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920880 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920882 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920885 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920889 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920891 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920894 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920896 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920899 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920901 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920904 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920906 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920909 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920912 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920915 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920917 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920920 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920922 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.933495 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920925 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920927 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920930 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920932 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920936 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920938 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920941 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920943 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920946 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920948 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920951 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920954 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920957 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920959 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920961 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920964 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920966 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920970 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920973 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.934074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920976 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920978 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920981 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920985 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920987 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920990 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920992 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920995 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920997 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.920999 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921002 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921004 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921007 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921009 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921012 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921014 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921016 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921020 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921023 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921026 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.934514 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921028 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921030 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921033 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921035 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921039 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921042 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921044 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921047 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921049 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921052 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921054 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921056 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921059 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921062 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921065 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921067 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921070 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921072 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921075 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921077 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.935002 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921079 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921082 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921084 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921087 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921090 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921094 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921098 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921102 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.921105 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.924920 5144 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.924934 5144 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.924953 5144 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.924958 5144 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925013 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925018 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925021 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925025 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925028 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925031 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.935455 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925034 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925036 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925039 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925042 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925046 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925051 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925054 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925057 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925060 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925062 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925065 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925067 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925070 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925072 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925075 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925077 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925079 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925081 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925084 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.936004 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925086 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925089 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925091 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925094 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925097 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925100 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925103 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925106 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925109 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925111 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925113 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925116 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925118 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925121 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925123 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925126 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925128 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925130 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925133 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925135 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.936435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925137 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925140 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925142 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925144 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925147 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925149 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925151 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925154 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925157 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925159 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925162 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925164 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925167 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925169 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925172 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925174 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925177 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925179 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925183 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.936960 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925185 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925188 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925190 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925193 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925195 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925197 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925200 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925202 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925204 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925207 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925209 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925211 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925214 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925216 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925218 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925221 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925223 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925225 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925227 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.937433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925230 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925232 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925234 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925237 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925240 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925242 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925244 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925247 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925249 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925252 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925254 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925257 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925260 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925262 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925264 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925267 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925270 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925272 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925274 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925277 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.937879 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925279 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925282 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925284 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.925311 5144 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925451 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925455 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925458 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925461 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925463 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925466 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925469 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925471 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925474 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925476 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925478 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925481 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.938347 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925484 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925486 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925489 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925491 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925494 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925497 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925499 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925502 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925504 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925507 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925509 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925511 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925514 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925516 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925518 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925521 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925523 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925525 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925528 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925530 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.938808 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925532 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925535 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925537 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925540 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925542 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925544 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925547 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925550 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925552 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925555 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925557 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925560 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925562 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925565 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925567 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925570 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925572 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925574 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925577 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.939292 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925579 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925581 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925584 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925586 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925589 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925592 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925594 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925596 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925599 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925601 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925604 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925606 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925608 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925612 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925616 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925619 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925623 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925625 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925641 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.939855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925644 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925646 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925649 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925651 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925654 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925656 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925659 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925661 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925663 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925666 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925668 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925670 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925673 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925676 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925679 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925682 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925684 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925687 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925690 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.940290 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925692 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925695 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925697 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925699 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925702 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925704 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925706 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925709 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925711 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925714 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925716 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925718 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925721 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925723 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925725 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925728 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.940774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925730 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.925753 5144 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925783 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925787 5144 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925789 5144 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925792 5144 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925794 5144 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925796 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925799 5144 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925801 5144 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925804 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925806 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925808 5144 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925811 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925814 5144 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925818 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 00:05:09.941144 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925821 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925824 5144 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925827 5144 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925829 5144 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925832 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925834 5144 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925837 5144 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925839 5144 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925844 5144 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925846 5144 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925848 5144 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925851 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925853 5144 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925856 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925858 5144 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925861 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925863 5144 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925866 5144 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925868 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925870 5144 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 00:05:09.941517 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925873 5144 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925875 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925878 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925880 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925882 5144 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925885 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925888 5144 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925893 5144 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925896 5144 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925900 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925904 5144 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925907 5144 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925910 5144 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925914 5144 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925918 5144 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925922 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925929 5144 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925933 5144 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925936 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 00:05:09.941999 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925939 5144 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925943 5144 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925946 5144 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925948 5144 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925951 5144 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925953 5144 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925956 5144 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925958 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925960 5144 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925963 5144 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925966 5144 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925968 5144 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925971 5144 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925974 5144 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925976 5144 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925979 5144 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925982 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925984 5144 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925986 5144 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925989 5144 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 00:05:09.942527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925991 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925994 5144 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925996 5144 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.925998 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926001 5144 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926004 5144 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926007 5144 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926009 5144 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926012 5144 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926016 5144 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926018 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926021 5144 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926023 5144 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926026 5144 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926029 5144 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926031 5144 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926034 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926037 5144 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926039 5144 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 00:05:09.943007 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926042 5144 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926044 5144 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926046 5144 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926049 5144 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926051 5144 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926054 5144 feature_gate.go:386] unrecognized feature gate: Example Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926056 5144 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926060 5144 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926064 5144 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926067 5144 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926070 5144 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926073 5144 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926075 5144 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:09.926077 5144 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.926724 5144 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928215 5144 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928268 5144 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928443 5144 server.go:1014] "Starting client certificate rotation" Nov 03 00:05:09.943572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928530 5144 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928579 5144 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:27 +0000 UTC" deadline="2028-04-12 11:22:44.184069202 +0000 UTC" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928605 5144 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12635h17m34.255465666s" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.928913 5144 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.929925 5144 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.931100 5144 log.go:25] "Validated CRI v1 runtime API" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.932722 5144 log.go:25] "Validated CRI v1 image API" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.933473 5144 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 00:05:09.944289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.935447 5144 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ff44378-c6a0-433d-97b7-c71042d66846:/dev/nvme0n1p4 ea3afd72-ac51-4520-8743-ce188ec0d9f0:/dev/nvme0n1p3] Nov 03 00:05:09.944728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.935467 5144 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e/userdata/shm major:0 minor:223 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7/userdata/shm major:0 minor:231 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03/userdata/shm major:0 minor:243 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18/userdata/shm major:0 minor:245 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e/userdata/shm major:0 minor:305 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e/userdata/shm major:0 minor:364 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82/userdata/shm major:0 minor:307 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce/userdata/shm major:0 minor:359 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0964e1df-d8ed-402a-831f-b8996a1236f3/volumes/kubernetes.io~projected/kube-api-access-jc8tf:{mountpoint:/var/lib/kubelet/pods/0964e1df-d8ed-402a-831f-b8996a1236f3/volumes/kubernetes.io~projected/kube-api-access-jc8tf major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0bbc3b7e-1648-4984-b4da-673a033aca22/volumes/kubernetes.io~projected/kube-api-access-dtb2k:{mountpoint:/var/lib/kubelet/pods/0bbc3b7e-1648-4984-b4da-673a033aca22/volumes/kubernetes.io~projected/kube-api-access-dtb2k major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d/volumes/kubernetes.io~projected/kube-api-access-n4qj2:{mountpoint:/var/lib/kubelet/pods/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d/volumes/kubernetes.io~projected/kube-api-access-n4qj2 major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a0eb70c-74c9-485b-a907-fa2830431a1d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/1a0eb70c-74c9-485b-a907-fa2830431a1d/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4/volumes/kubernetes.io~projected/kube-api-access-9wr9s:{mountpoint:/var/lib/kubelet/pods/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4/volumes/kubernetes.io~projected/kube-api-access-9wr9s major:0 minor:323 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1edd18e2-9aa6-4708-98e9-773c0a38a368/volumes/kubernetes.io~projected/kube-api-access-j6jzx:{mountpoint:/var/lib/kubelet/pods/1edd18e2-9aa6-4708-98e9-773c0a38a368/volumes/kubernetes.io~projected/kube-api-access-j6jzx major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6/volumes/kubernetes.io~projected/kube-api-access-779m4:{mountpoint:/var/lib/kubelet/pods/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6/volumes/kubernetes.io~projected/kube-api-access-779m4 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2ee7c64c-4945-469a-8d27-cfb702aabf50/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2ee7c64c-4945-469a-8d27-cfb702aabf50/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:299 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~projected/kube-api-access-9t5vr:{mountpoint:/var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~projected/kube-api-access-9t5vr major:0 minor:209 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~secret/default-certificate major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~secret/stats-auth major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59/volumes/kubernetes.io~projected/kube-api-access-trkh5:{mountpoint:/var/lib/kubelet/pods/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59/volumes/kubernetes.io~projected/kube-api-access-trkh5 major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/35ce84ec-29fd-49ab-9c7a-092e99f093b8/volumes/kubernetes.io~projected/kube-api-access-jsx2q:{mountpoint:/var/lib/kubelet/pods/35ce84ec-29fd-49ab-9c7a-092e99f093b8/volumes/kubernetes.io~projected/kube-api-access-jsx2q major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/66a8deff-4567-470f-98d0-bec9ea590d9a/volumes/kubernetes.io~projected/kube-api-access-hb28b:{mountpoint:/var/lib/kubelet/pods/66a8deff-4567-470f-98d0-bec9ea590d9a/volumes/kubernetes.io~projected/kube-api-access-hb28b major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1/volumes/kubernetes.io~projected/kube-api-access-qmqm4:{mountpoint:/var/lib/kubelet/pods/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1/volumes/kubernetes.io~projected/kube-api-access-qmqm4 major:0 minor:304 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/745ded03-fed2-450a-bb70-8a78b9d3200d/volumes/kubernetes.io~projected/kube-api-access-kmqlg:{mountpoint:/var/lib/kubelet/pods/745ded03-fed2-450a-bb70-8a78b9d3200d/volumes/kubernetes.io~projected/kube-api-access-kmqlg major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/745ded03-fed2-450a-bb70-8a78b9d3200d/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/745ded03-fed2-450a-bb70-8a78b9d3200d/volumes/kubernetes.io~secret/serving-cert major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/772d8d93-5c01-4e16-832e-51dbcf7191bd/volumes/kubernetes.io~projected/kube-api-access-f4hnc:{mountpoint:/var/lib/kubelet/pods/772d8d93-5c01-4e16-832e-51dbcf7191bd/volumes/kubernetes.io~projected/kube-api-access-f4hnc major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5244e9-0e47-4a6d-a27d-24ce8455a537/volumes/kubernetes.io~projected/kube-api-access-447n8:{mountpoint:/var/lib/kubelet/pods/7c5244e9-0e47-4a6d-a27d-24ce8455a537/volumes/kubernetes.io~projected/kube-api-access-447n8 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c5244e9-0e47-4a6d-a27d-24ce8455a537/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/7c5244e9-0e47-4a6d-a27d-24ce8455a537/volumes/kubernetes.io~secret/serving-cert major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7d665bcd-2a5f-45c8-802a-370fc4a4aefd/volumes/kubernetes.io~projected/kube-api-access-54cx9:{mountpoint:/var/lib/kubelet/pods/7d665bcd-2a5f-45c8-802a-370fc4a4aefd/volumes/kubernetes.io~projected/kube-api-access-54cx9 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82a02683-66a4-43a5-9d32-c53f72d1f613/volumes/kubernetes.io~projected/kube-api-access-vfhs5:{mountpoint:/var/lib/kubelet/pods/82a02683-66a4-43a5-9d32-c53f72d1f613/volumes/kubernetes.io~projected/kube-api-access-vfhs5 major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/volumes/kubernetes.io~projected/kube-api-access-x9fbr:{mountpoint:/var/lib/kubelet/pods/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/volumes/kubernetes.io~projected/kube-api-access-x9fbr major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/volumes/kubernetes.io~secret/cert major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/88200eac-4b64-4019-b733-335740bfe1e5/volumes/kubernetes.io~projected/kube-api-access-pd9lf:{mountpoint:/var/lib/kubelet/pods/88200eac-4b64-4019-b733-335740bfe1e5/volumes/kubernetes.io~projected/kube-api-access-pd9lf major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9425f6e8-123b-4acb-a3d1-5ccd19761401/volumes/kubernetes.io~projected/kube-api-access-dqsbt:{mountpoint:/var/lib/kubelet/pods/9425f6e8-123b-4acb-a3d1-5ccd19761401/volumes/kubernetes.io~projected/kube-api-access-dqsbt major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804/volumes/kubernetes.io~projected/kube-api-access-krrk7:{mountpoint:/var/lib/kubelet/pods/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804/volumes/kubernetes.io~projected/kube-api-access-krrk7 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volumes/kubernetes.io~projected/kube-api-access-4bd78:{mountpoint:/var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volumes/kubernetes.io~projected/kube-api-access-4bd78 major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ca0220e0-47de-45fb-bbb6-11ed27994448/volumes/kubernetes.io~projected/kube-api-access-wgjgg:{mountpoint:/var/lib/kubelet/pods/ca0220e0-47de-45fb-bbb6-11ed27994448/volumes/kubernetes.io~projected/kube-api-access-wgjgg major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d7e0f3c7-a695-4b00-8178-1955b1159469/volumes/kubernetes.io~projected/kube-api-access-thvvt:{mountpoint:/var/lib/kubelet/pods/d7e0f3c7-a695-4b00-8178-1955b1159469/volumes/kubernetes.io~projected/kube-api-access-thvvt major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~empty-dir/tmp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~projected/kube-api-access-5r8tz:{mountpoint:/var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~projected/kube-api-access-5r8tz major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~projected/kube-api-access-82m2t:{mountpoint:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~projected/kube-api-access-82m2t major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~projected/kube-api-access-hgpgv:{mountpoint:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~projected/kube-api-access-hgpgv major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8fc0eda-47b5-49fb-86bd-3e23f563c46b/volumes/kubernetes.io~projected/kube-api-access-bd7n4:{mountpoint:/var/lib/kubelet/pods/e8fc0eda-47b5-49fb-86bd-3e23f563c46b/volumes/kubernetes.io~projected/kube-api-access-bd7n4 major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed231fcb-17c8-4887-b2b3-1fb1c64eb442/volumes/kubernetes.io~projected/kube-api-access-b7w8d:{mountpoint:/var/lib/kubelet/pods/ed231fcb-17c8-4887-b2b3-1fb1c64eb442/volumes/kubernetes.io~projected/kube-api-access-b7w8d major:0 minor:356 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0f94dde-4d2f-4969-b7ce-357282dc6529/volumes/kubernetes.io~projected/kube-api-access-kw657:{mountpoint:/var/lib/kubelet/pods/f0f94dde-4d2f-4969-b7ce-357282dc6529/volumes/kubernetes.io~projected/kube-api-access-kw657 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd/volumes/kubernetes.io~projected/kube-api-access-dgq4q:{mountpoint:/var/lib/kubelet/pods/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd/volumes/kubernetes.io~projected/kube-api-access-dgq4q major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd/volumes/kubernetes.io~secret/signing-key major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f58b544f-c636-4c93-9e1c-9a817590d307/volumes/kubernetes.io~projected/kube-api-access-rpwkv:{mountpoint:/var/lib/kubelet/pods/f58b544f-c636-4c93-9e1c-9a817590d307/volumes/kubernetes.io~projected/kube-api-access-rpwkv major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f58b544f-c636-4c93-9e1c-9a817590d307/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/f58b544f-c636-4c93-9e1c-9a817590d307/volumes/kubernetes.io~secret/serving-cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba48a24-ec24-4b58-820c-c69010288372/volumes/kubernetes.io~projected/kube-api-access-gvcfs:{mountpoint:/var/lib/kubelet/pods/fba48a24-ec24-4b58-820c-c69010288372/volumes/kubernetes.io~projected/kube-api-access-gvcfs major:0 minor:62 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/00195daa4a73bba9ff71172c4b3ca68c160d29b24202688abfe69fa7ded38f5a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/294822872c7400668979e2e593f11abd1c849d8ddd518563174ca0186f7c8df4/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/99c349ea79325e178c3939859c2dfbf797c2de5046b787af86d3ddf1bbb15a2a/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-109:{mountpoint:/var/lib/containers/storage/overlay/c2fe44af4859f3e05a3de7493fe20b82dc8de791362d07941a98eae83ac62005/merged major:0 minor:109 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/b4cfacac1bd9e214e54741e6af6e1996a5fa51b934c8084893961d8e907a20f1/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/8c05e7b9641fb4f377024c00a6e06e7881281aeab6824159c20edca5d2df7143/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f82ce65ad38f52a52e2da51b7ede72c23e0a9e540e19ccd484ed775a38d50eec/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/9a25e31895bdf895d5af5f69dc8d35dfbf54400d65de8ca929d941c865c2831f/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/5fe4c45bd6fe79a8c6d653098a0cb8eb08f2530ea90b288dd084606196d7f6dc/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/f8f6c41f1321dce101244b8dbb61561fe58c2dfb9040bfb7029897cac1076e2c/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/8769fc02615a28104c63c57800f6e3c5b50a49e302f07152a23fd6ec7e0031ba/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/46315edd207180873e930f6508b2ab200edd8ed52b6e09d45569e635aef9374e/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/9ccbbfc049734f165127f053e1de59332ab38a1ba2f3f33abe0979f58b0f1861/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d2d1884f5286dec088abcd7c8de50b3027e346de9a206d64083e7e6ed09026aa/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/95094156158c9ddc0436ace16ea86f7bf41772a1122ee0a97684dbd60c94c91d/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/03c0102b8a048853f58de333fb89c118bc342901ba1f178441c34fd5f11d7092/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/708251b0fd025f129665894f79463ffe4d64020ba310c4db449722f7b56a2ba1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/3eb1ef418caeb5dd310f3042a1071b4050a03b95d5c2738135269351a531b98b/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c73e8d96cf0440bc35ea92fceb36ac25c5c15cd9a04e84d161ae30e525a0e24e/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/f8754ab5b14f29a4534092725f57ccba6632f54e840fd3fbeef02ddec8333652/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/8ebf991901a0510d854718371afbd24219b15ad45e96fe3d823e0e3b98df7f00/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/186db623443ed3dee82b8590c2b1edb283081ae3f21b76ea195a20aaa518ae8f/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/b10e2a7d3bb3910589a098b43896773cbf3f214e1ed988899bf67f9f4cd968db/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/564807eb56f16b94c0bb14a8d2ad1578a1ec153ddb7ebcfd6afee0cdc8b25035/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/e6e89a95fac4e1f022acad753f243f6a6d37b29834befb11e24a7e7aa4b420d8/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-259:{mountpoint:/var/lib/containers/storage/overlay/6678b29a33b7b9c766f701158cb6d605cec2a7c8891ebf95791f64e03c2956e9/merged major:0 minor:259 fsType:overlay blockSize:0} overlay_0-261:{mountpoint:/var/lib/containers/storage/overlay/3039cf72535b2d1fbbab53072bae764ebc84a049bf10d7812246b5c287fa1585/merged major:0 minor:261 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/181513813cdafb4150baea57458168b2b76c4ddbc512e6e815c9ae4e2ad01cbe/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/d4777ebd319f75b1969842a99c879969037b8a26e563f9d9c0e0999b3c986598/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-267:{mountpoint:/var/lib/containers/storage/overlay/8f80dda9ff721f46742cdec0f890e48a2245ab0fec9456129b470881c17af42a/merged major:0 minor:267 fsType:overlay blockSize:0} overlay_0-277:{mountpoint:/var/lib/containers/storage/overlay/eae0084a9f09736047a47425e8ecafae384987f763e5f6dcf0d09839948a5e22/merged major:0 minor:277 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/1e3775eacf624133414fb25a3c324a7db85dcb617c379677c86dc881807abe5f/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-281:{mountpoint:/var/lib/containers/storage/overlay/8ad758c869d14db92459e2e584fbf301d01672d119998678a32f28fc647edb1b/merged major:0 minor:281 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/43e4ba14ceca1efa3b9bcd7ddf396acc13f2652d6eabbb42bcbaf858a1ea5cce/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/a3e3018c7cff3ecca8e636b4470ee414e366dc64bd0d68f259b76b82cbb61687/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/228db98c65089eccefab39957ae760af5d8c722e45a6fd46c1450b7f3419cd09/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/e2a56771a5ca6c143e2c9255565f3be0f5f0c2e9718de0db39fb45419e9a9880/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-309:{mountpoint:/var/lib/containers/storage/overlay/414f6bf107d3566d6e42f27ac336889a6151aed0670298b978b28ce5df54a59c/merged major:0 minor:309 fsType:overlay blockSize:0} overlay_0-311:{mountpoint:/var/lib/containers/storage/overlay/2102a90ba3727c76e50d0b92394a402994549d54678807208b5cee7e0abb0003/merged major:0 minor:311 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/d034187379f3419db7f8a04aecdf235d9a8f21842de20ce2d03ed1fccce68308/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/52bb9e09e19e4f4131dcf6713ad1bfa92a46a8dad6d719f11813b66e82d05354/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/1bc2048780a83bdd86dadb58ac6c6509132ec20b88dd0cbb0df60e238c33d699/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/2f7cfd05cb58d3cfc70678c39ac2c38eeb70f7c4ce8803a79ea82ee9186743e2/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/00b0ddb3cf4ed9af1f6c5587043d43fec8fd70fb4a838ea02da1bb5ae49df020/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/085c77f8b7be52787b227899bdb3045af4b83623be96b2ed9b5d4d3625ee631f/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/07765eb7400183353408421f69de477d5ac5b55f4e6546d4fc34186fc4a53eba/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/71f1430928ce86fb7ac940bdef303c19fcfa6753f5f24b5ef90da16fa564cc0a/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/5244b41202fff5cee60a425da10199428914abd5d9e0bb906272e83772081cf6/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-369:{mountpoint:/var/lib/containers/storage/overlay/4152368cd1299602383bd819b0131d537dcd413d74aadfe606f0c5e499399fef/merged major:0 minor:369 fsType:overlay blockSize:0} overlay_0-375:{mountpoint:/var/lib/containers/storage/overlay/17beec0aa12f5345eeff8ae27b841e509d078ce1ef028ef3206619dcc919db33/merged major:0 minor:375 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/ab13dbd1da485c0e5d4c72b672e88e730673767635726e616a99976052b56daf/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/6aee8fb6f67fed672970418308658d8de6980a50cb9813d6942c95d0ba069860/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/8af3a108126b7b1c574edc28ce009c8622899df708f04c40b73bcef1cf956ff9/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/ae4dd7d6f04abd819852a7885a72b20cd23e9905b1f1f5ac2984af28bcfa4e1b/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/37b6d6fb13a990e2c844beb2f1296e959fa2e19560ed0955d3f3d1d5264f268b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/bf2e9aafff44ed4bc1c94c9e39d6ec3bdcc3a9a314990cb3248796765e9b060c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/ebe111005ada1bfd47f711604153d995cd15a6ee60fd370185741b096f480c1a/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4b2f76668041345187d6541801285f2fa5696b201934f59300934392a78ba3f4/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/30e488d0074b435b515217a0d093d5ecf4621ab839ec911d8a6503f0450b02bc/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/86d43be4166a54b3cfb1769be6d2804558b1057b9c1b86bd5d7576c081b2f99d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/973e91bb4862738c249857d1501a522a98cb7aa09ccc80e90d0709171628afde/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/c684c4aafbc5913cd56caeac4d7e467d8d827334dcee8941d94b6720eaa0eda1/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9dbfed2aecc501646d840b69185210ec617755400477ce1014957b1250f94a76/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/f3c2f8ae97699bbeb7098bfd55ce148b2b7736c4bd789e6bf2fd660a3ff46718/merged major:0 minor:98 fsType:overlay blockSize:0}] Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.943012 5144 manager.go:217] Machine: {Timestamp:2026-11-03 00:05:09.942552117 +0000 UTC m=+0.088246563 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3167857 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec269f13b1ded54a6f3092b0b2fd822c SystemUUID:ec269f13-b1de-d54a-6f30-92b0b2fd822c BootID:da0b9dc2-04a3-4244-9c00-eae3076f2fe1 Filesystems:[{Device:/var/lib/kubelet/pods/ca0220e0-47de-45fb-bbb6-11ed27994448/volumes/kubernetes.io~projected/kube-api-access-wgjgg DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f58b544f-c636-4c93-9e1c-9a817590d307/volumes/kubernetes.io~projected/kube-api-access-rpwkv DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/745ded03-fed2-450a-bb70-8a78b9d3200d/volumes/kubernetes.io~projected/kube-api-access-kmqlg DeviceMajor:0 DeviceMinor:235 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e8fc0eda-47b5-49fb-86bd-3e23f563c46b/volumes/kubernetes.io~projected/kube-api-access-bd7n4 DeviceMajor:0 DeviceMinor:242 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f82e74bc7b111a61f80f1e050b8ecdf026000555ce5ad6bb7ba2192e5070707/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volumes/kubernetes.io~projected/kube-api-access-4bd78 DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~projected/kube-api-access-5r8tz DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-267 DeviceMajor:0 DeviceMinor:267 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6294367b748c804d621b730da7e577ba4c49fa264908da1a96e7745f6a1680ce/userdata/shm DeviceMajor:0 DeviceMinor:359 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa842df183f000ce21ac9b4e16ddd5d5fccdcefd33388c0496571a90120d6201/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59/volumes/kubernetes.io~projected/kube-api-access-trkh5 DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/35ce84ec-29fd-49ab-9c7a-092e99f093b8/volumes/kubernetes.io~projected/kube-api-access-jsx2q DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4/volumes/kubernetes.io~projected/kube-api-access-9wr9s DeviceMajor:0 DeviceMinor:323 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/ed231fcb-17c8-4887-b2b3-1fb1c64eb442/volumes/kubernetes.io~projected/kube-api-access-b7w8d DeviceMajor:0 DeviceMinor:356 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/846d1e818d3538e1cadcd1317fb5bcc27c0e397663e18260e2e83f16bebf57d8/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1edd18e2-9aa6-4708-98e9-773c0a38a368/volumes/kubernetes.io~projected/kube-api-access-j6jzx DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/82a02683-66a4-43a5-9d32-c53f72d1f613/volumes/kubernetes.io~projected/kube-api-access-vfhs5 DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5f1ef120d40862f3e588c1a412f5fcc48389e130c517a3d2cbb18be806d39c94/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/187c3cc72214fd2b86e310fb8727727e299bc85b3cb266ece022fa69d183ec03/userdata/shm DeviceMajor:0 DeviceMinor:243 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c76d3461b4035caf76829320a4e8b5a58545bd7be8f5da8f64e63a1695cf89b4/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1a0eb70c-74c9-485b-a907-fa2830431a1d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/volumes/kubernetes.io~projected/kube-api-access-x9fbr DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-311 DeviceMajor:0 DeviceMinor:311 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/534b44a5820ad89abb3db75980c86306aced80280c053fb89f531092fb79c06e/userdata/shm DeviceMajor:0 DeviceMinor:364 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fba48a24-ec24-4b58-820c-c69010288372/volumes/kubernetes.io~projected/kube-api-access-gvcfs DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/707d34bc3c7eeebc51444f1688662704b63e193305f1a78c3f40d09ab823fafc/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fd911a09bfa688976c99da0a743c5f051fd21aa9fdfb7ce078bcf81bc70390d3/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a93829e89204daba3796b946c7f1bc07c6871eba905fad889838912a5a3a63eb/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-277 DeviceMajor:0 DeviceMinor:277 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-281 DeviceMajor:0 DeviceMinor:281 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5244e9-0e47-4a6d-a27d-24ce8455a537/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:203 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/204977683e1ae3b2ffa4f1d31bbc078d893f57c8d864dc3c8f7eb9c9904c7b6e/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/88200eac-4b64-4019-b733-335740bfe1e5/volumes/kubernetes.io~projected/kube-api-access-pd9lf DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~projected/kube-api-access-hgpgv DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:371 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-109 DeviceMajor:0 DeviceMinor:109 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0964e1df-d8ed-402a-831f-b8996a1236f3/volumes/kubernetes.io~projected/kube-api-access-jc8tf DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-309 DeviceMajor:0 DeviceMinor:309 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01d528dcba73de3be65e3436ca121f5e96fa966dda170c820eb7d95ebcc3717e/userdata/shm DeviceMajor:0 DeviceMinor:223 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-375 DeviceMajor:0 DeviceMinor:375 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804/volumes/kubernetes.io~projected/kube-api-access-krrk7 DeviceMajor:0 DeviceMinor:210 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0bbc3b7e-1648-4984-b4da-673a033aca22/volumes/kubernetes.io~projected/kube-api-access-dtb2k DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/745ded03-fed2-450a-bb70-8a78b9d3200d/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~projected/kube-api-access-82m2t DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1975b46199b1c2f38f2d23cf16dc3b0518e4867413dcdcd68650f6bf0f918d18/userdata/shm DeviceMajor:0 DeviceMinor:245 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/772d8d93-5c01-4e16-832e-51dbcf7191bd/volumes/kubernetes.io~projected/kube-api-access-f4hnc DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef9b7669ec214af33ff10863027baadc840b827b95171b5016c7509fe426fdcc/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9425f6e8-123b-4acb-a3d1-5ccd19761401/volumes/kubernetes.io~projected/kube-api-access-dqsbt DeviceMajor:0 DeviceMinor:328 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/094517d20acbdd7f846730b42bba40005ae396f1e95147b0a523cb0d75b28c31/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/deb16adf-e150-4c31-8b3f-6509c0b919e3/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4339c058837ffcc51c29bc262d89ad9d13523b400560d63b28722a9c63046fe/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d7e0f3c7-a695-4b00-8178-1955b1159469/volumes/kubernetes.io~projected/kube-api-access-thvvt DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f58b544f-c636-4c93-9e1c-9a817590d307/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd/volumes/kubernetes.io~projected/kube-api-access-dgq4q DeviceMajor:0 DeviceMinor:372 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/db711a121172b6ee78f161e87ae9f4810a7e5ef5713bd94a5750db14b29c1bbc/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-259 DeviceMajor:0 DeviceMinor:259 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:239 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4cc74a5e22417293f5104a11fae9303ec0628b9492a03de35c184e34694afc7e/userdata/shm DeviceMajor:0 DeviceMinor:305 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f0f94dde-4d2f-4969-b7ce-357282dc6529/volumes/kubernetes.io~projected/kube-api-access-kw657 DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2eb3cf3e5af21a41be23d8c01980bb50c5155ea87f84fa8791ad02ddc63bbf40/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-261 DeviceMajor:0 DeviceMinor:261 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2ee7c64c-4945-469a-8d27-cfb702aabf50/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:299 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df3093e792a0c098c63057008c3bb7ea01792a3897554aba316a243abbd58fc3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6/volumes/kubernetes.io~projected/kube-api-access-779m4 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/098dc10d1d62e8079b3e3bee818ccecdf02fedea7c7626709e2d7d9b292864b7/userdata/shm DeviceMajor:0 DeviceMinor:231 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-369 DeviceMajor:0 DeviceMinor:369 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/31e371fb-f718-43d0-bf0f-d909838ac2d0/volumes/kubernetes.io~projected/kube-api-access-9t5vr DeviceMajor:0 DeviceMinor:209 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d/volumes/kubernetes.io~projected/kube-api-access-n4qj2 DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/66a8deff-4567-470f-98d0-bec9ea590d9a/volumes/kubernetes.io~projected/kube-api-access-hb28b DeviceMajor:0 DeviceMinor:351 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7d665bcd-2a5f-45c8-802a-370fc4a4aefd/volumes/kubernetes.io~projected/kube-api-access-54cx9 DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7c5244e9-0e47-4a6d-a27d-24ce8455a537/volumes/kubernetes.io~projected/kube-api-access-447n8 DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:225 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55f6e70dc3c04975d84763da841dff1d6218c0f60ca506872d84154f81632e82/userdata/shm DeviceMajor:0 DeviceMinor:307 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:204 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1/volumes/kubernetes.io~projected/kube-api-access-qmqm4 DeviceMajor:0 DeviceMinor:304 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01d528dcba73de3 MacAddress:fa:5f:bf:89:cd:fd Speed:10000 Mtu:8901} {Name:094517d20acbdd7 MacAddress:52:80:d7:9b:80:1a Speed:10000 Mtu:8901} {Name:098dc10d1d62e80 MacAddress:1a:58:aa:58:ae:72 Speed:10000 Mtu:8901} {Name:187c3cc72214fd2 MacAddress:1a:fa:4d:de:f6:6a Speed:10000 Mtu:8901} {Name:1975b46199b1c2f MacAddress:ca:bf:3f:0a:c3:11 Speed:10000 Mtu:8901} {Name:204977683e1ae3b MacAddress:9a:03:e4:c6:5d:1e Speed:10000 Mtu:8901} {Name:26d3a6eade45c27 MacAddress:6e:f7:60:b1:b1:57 Speed:10000 Mtu:8901} {Name:4cc74a5e2241729 MacAddress:6e:3d:6f:b9:00:1d Speed:10000 Mtu:8901} {Name:534b44a5820ad89 MacAddress:b6:38:f7:5e:1c:6a Speed:10000 Mtu:8901} {Name:55f6e70dc3c0497 MacAddress:f6:73:dd:80:6e:5a Speed:10000 Mtu:8901} {Name:5f1ef120d40862f MacAddress:36:12:8d:05:89:0e Speed:10000 Mtu:8901} {Name:6294367b748c804 MacAddress:26:ac:18:60:25:c6 Speed:10000 Mtu:8901} {Name:6f82e74bc7b111a MacAddress:62:d8:16:38:ae:72 Speed:10000 Mtu:8901} {Name:846d1e818d3538e MacAddress:ca:6d:77:82:cb:5e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:03:62:05:11:97 Speed:0 Mtu:9001} {Name:br-int MacAddress:56:0d:d8:87:ec:23 Speed:0 Mtu:8901} {Name:c76d3461b4035ca MacAddress:22:a1:0d:58:41:08 Speed:10000 Mtu:8901} {Name:db711a121172b6e MacAddress:36:8a:55:1a:11:6b Speed:10000 Mtu:8901} {Name:ea08a296039010f MacAddress:92:58:d1:15:86:5c Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:03:62:05:11:97 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:c6:d1:c9:9e:c9:f4 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:b2:63:95:4f:b9:52 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.943912 5144 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.943985 5144 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.944404 5144 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.944428 5144 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-143-162.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.944893 5144 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.944904 5144 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.944922 5144 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:05:09.945007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.944942 5144 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945032 5144 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945183 5144 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945291 5144 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945323 5144 kubelet.go:493] "Attempting to sync node with API server" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945335 5144 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945347 5144 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945355 5144 kubelet.go:405] "Adding apiserver pod source" Nov 03 00:05:09.945397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.945363 5144 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 00:05:09.946249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.946235 5144 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:05:09.946249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.946248 5144 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:05:09.946751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.946734 5144 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 00:05:09.948790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.948777 5144 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 00:05:09.948790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.948790 5144 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 00:05:09.948891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.948882 5144 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 00:05:09.949065 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949054 5144 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 00:05:09.949098 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949082 5144 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 00:05:09.949179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949170 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949184 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949190 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949196 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949202 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949208 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949213 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 00:05:09.949218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949218 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 00:05:09.949469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949225 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 00:05:09.949469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949236 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 00:05:09.949469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949249 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 00:05:09.949469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949261 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 00:05:09.949469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949290 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 00:05:09.949469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.949297 5144 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 00:05:09.952693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.952680 5144 server.go:1263] "Started kubelet" Nov 03 00:05:09.952768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.952734 5144 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 00:05:09.952834 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.952777 5144 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 00:05:09.952888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.952860 5144 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 00:05:09.953206 ip-10-0-143-162 systemd[1]: Started Kubernetes Kubelet. Nov 03 00:05:09.953318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.953293 5144 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 00:05:09.953963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.953946 5144 server.go:317] "Adding debug handlers to kubelet server" Nov 03 00:05:09.958404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.958385 5144 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 00:05:09.958591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.958572 5144 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 00:05:09.958778 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.958663 5144 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:29 +0000 UTC" deadline="2028-04-03 11:25:34.420220455 +0000 UTC" Nov 03 00:05:09.958870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.958781 5144 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12419h20m24.461444999s" Nov 03 00:05:09.959016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.958996 5144 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 00:05:09.959016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.959016 5144 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 00:05:09.959173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.959148 5144 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 00:05:09.960879 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:09.959719 5144 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:05:09.962999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.962419 5144 factory.go:55] Registering systemd factory Nov 03 00:05:09.962999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.962442 5144 factory.go:223] Registration of the systemd container factory successfully Nov 03 00:05:09.963564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963529 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets" seLinuxMountContext="" Nov 03 00:05:09.963610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963570 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a0eb70c-74c9-485b-a907-fa2830431a1d" volumeName="kubernetes.io/secret/1a0eb70c-74c9-485b-a907-fa2830431a1d-agent-certs" seLinuxMountContext="" Nov 03 00:05:09.963610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963583 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="745ded03-fed2-450a-bb70-8a78b9d3200d" volumeName="kubernetes.io/secret/745ded03-fed2-450a-bb70-8a78b9d3200d-serving-cert" seLinuxMountContext="" Nov 03 00:05:09.963610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963594 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5244e9-0e47-4a6d-a27d-24ce8455a537" volumeName="kubernetes.io/secret/7c5244e9-0e47-4a6d-a27d-24ce8455a537-serving-cert" seLinuxMountContext="" Nov 03 00:05:09.963610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963605 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d665bcd-2a5f-45c8-802a-370fc4a4aefd" volumeName="kubernetes.io/projected/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-kube-api-access-54cx9" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963616 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d7e0f3c7-a695-4b00-8178-1955b1159469" volumeName="kubernetes.io/projected/d7e0f3c7-a695-4b00-8178-1955b1159469-kube-api-access-thvvt" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963652 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963664 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963676 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29e4053-bef9-482f-8fe1-44a0d9bd4dfd" volumeName="kubernetes.io/projected/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-kube-api-access-dgq4q" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963687 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d" volumeName="kubernetes.io/configmap/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-telemetry-config" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963698 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59" volumeName="kubernetes.io/configmap/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-config" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963714 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4" volumeName="kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-script-lib" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963725 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb16adf-e150-4c31-8b3f-6509c0b919e3" volumeName="kubernetes.io/projected/deb16adf-e150-4c31-8b3f-6509c0b919e3-kube-api-access-5r8tz" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963736 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963749 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963760 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58b544f-c636-4c93-9e1c-9a817590d307" volumeName="kubernetes.io/projected/f58b544f-c636-4c93-9e1c-9a817590d307-kube-api-access-rpwkv" seLinuxMountContext="" Nov 03 00:05:09.963775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963773 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c44bcac-ee48-48af-b5b4-99f2ce0e81d6" volumeName="kubernetes.io/projected/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-kube-api-access-779m4" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963785 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82a02683-66a4-43a5-9d32-c53f72d1f613" volumeName="kubernetes.io/configmap/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-config" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963795 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82a02683-66a4-43a5-9d32-c53f72d1f613" volumeName="kubernetes.io/projected/82a02683-66a4-43a5-9d32-c53f72d1f613-kube-api-access-vfhs5" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963806 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb16adf-e150-4c31-8b3f-6509c0b919e3" volumeName="kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-tmp" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963817 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e371fb-f718-43d0-bf0f-d909838ac2d0" volumeName="kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-default-certificate" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963828 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5244e9-0e47-4a6d-a27d-24ce8455a537" volumeName="kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-trusted-ca" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963838 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9425f6e8-123b-4acb-a3d1-5ccd19761401" volumeName="kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963849 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963861 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1edd18e2-9aa6-4708-98e9-773c0a38a368" volumeName="kubernetes.io/projected/1edd18e2-9aa6-4708-98e9-773c0a38a368-kube-api-access-j6jzx" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963872 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="745ded03-fed2-450a-bb70-8a78b9d3200d" volumeName="kubernetes.io/projected/745ded03-fed2-450a-bb70-8a78b9d3200d-kube-api-access-kmqlg" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963883 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ca0220e0-47de-45fb-bbb6-11ed27994448" volumeName="kubernetes.io/projected/ca0220e0-47de-45fb-bbb6-11ed27994448-kube-api-access-wgjgg" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963894 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963908 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963919 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba48a24-ec24-4b58-820c-c69010288372" volumeName="kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963930 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4" volumeName="kubernetes.io/projected/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-kube-api-access-4bd78" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963940 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2ee7c64c-4945-469a-8d27-cfb702aabf50" volumeName="kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963950 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59" volumeName="kubernetes.io/secret/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-serving-cert" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963960 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8505e4f1-35bd-4ff0-ad33-2febfff8fbf7" volumeName="kubernetes.io/projected/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-kube-api-access-x9fbr" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963971 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ae7af53a-1838-409e-b5f3-6aa5c688dd55" volumeName="kubernetes.io/configmap/ae7af53a-1838-409e-b5f3-6aa5c688dd55-nginx-conf" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963981 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.963991 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4" volumeName="kubernetes.io/secret/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964002 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d" volumeName="kubernetes.io/projected/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-kube-api-access-n4qj2" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964013 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88200eac-4b64-4019-b733-335740bfe1e5" volumeName="kubernetes.io/empty-dir/88200eac-4b64-4019-b733-335740bfe1e5-tmp-dir" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964024 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9425f6e8-123b-4acb-a3d1-5ccd19761401" volumeName="kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964034 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58b544f-c636-4c93-9e1c-9a817590d307" volumeName="kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-service-ca-bundle" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964045 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e371fb-f718-43d0-bf0f-d909838ac2d0" volumeName="kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-stats-auth" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964056 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1" volumeName="kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964067 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5244e9-0e47-4a6d-a27d-24ce8455a537" volumeName="kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-config" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964077 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c5244e9-0e47-4a6d-a27d-24ce8455a537" volumeName="kubernetes.io/projected/7c5244e9-0e47-4a6d-a27d-24ce8455a537-kube-api-access-447n8" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964087 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" volumeName="kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964098 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2c44bcac-ee48-48af-b5b4-99f2ce0e81d6" volumeName="kubernetes.io/configmap/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-iptables-alerter-script" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964109 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f94dde-4d2f-4969-b7ce-357282dc6529" volumeName="kubernetes.io/projected/f0f94dde-4d2f-4969-b7ce-357282dc6529-kube-api-access-kw657" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964119 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58b544f-c636-4c93-9e1c-9a817590d307" volumeName="kubernetes.io/secret/f58b544f-c636-4c93-9e1c-9a817590d307-serving-cert" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964131 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a0eb70c-74c9-485b-a907-fa2830431a1d" volumeName="kubernetes.io/configmap/1a0eb70c-74c9-485b-a907-fa2830431a1d-konnectivity-ca" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964141 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="88200eac-4b64-4019-b733-335740bfe1e5" volumeName="kubernetes.io/projected/88200eac-4b64-4019-b733-335740bfe1e5-kube-api-access-pd9lf" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964152 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964163 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964174 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8fc0eda-47b5-49fb-86bd-3e23f563c46b" volumeName="kubernetes.io/projected/e8fc0eda-47b5-49fb-86bd-3e23f563c46b-kube-api-access-bd7n4" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964185 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29e4053-bef9-482f-8fe1-44a0d9bd4dfd" volumeName="kubernetes.io/secret/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-key" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964197 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" volumeName="kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964216 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="df250e4d-45bd-4a42-9faa-a2ed51f7589b" volumeName="kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964227 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba48a24-ec24-4b58-820c-c69010288372" volumeName="kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964239 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc9f90a-15a5-41ec-9ad8-90969c74c8d4" volumeName="kubernetes.io/projected/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-api-access-9wr9s" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964249 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4" volumeName="kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-env-overrides" seLinuxMountContext="" Nov 03 00:05:09.964240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964260 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4" volumeName="kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-config" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964272 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0f94dde-4d2f-4969-b7ce-357282dc6529" volumeName="kubernetes.io/configmap/f0f94dde-4d2f-4969-b7ce-357282dc6529-serviceca" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964283 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f29e4053-bef9-482f-8fe1-44a0d9bd4dfd" volumeName="kubernetes.io/configmap/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-cabundle" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964295 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58b544f-c636-4c93-9e1c-9a817590d307" volumeName="kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-tmp" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964307 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58b544f-c636-4c93-9e1c-9a817590d307" volumeName="kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-snapshots" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964318 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31e371fb-f718-43d0-bf0f-d909838ac2d0" volumeName="kubernetes.io/projected/31e371fb-f718-43d0-bf0f-d909838ac2d0-kube-api-access-9t5vr" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964327 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59" volumeName="kubernetes.io/projected/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-kube-api-access-trkh5" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964339 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f58b544f-c636-4c93-9e1c-9a817590d307" volumeName="kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-trusted-ca-bundle" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964350 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba48a24-ec24-4b58-820c-c69010288372" volumeName="kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-binary-copy" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964360 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1cc9f90a-15a5-41ec-9ad8-90969c74c8d4" volumeName="kubernetes.io/empty-dir/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-data-volume" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964399 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="66a8deff-4567-470f-98d0-bec9ea590d9a" volumeName="kubernetes.io/projected/66a8deff-4567-470f-98d0-bec9ea590d9a-kube-api-access-hb28b" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964411 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="772d8d93-5c01-4e16-832e-51dbcf7191bd" volumeName="kubernetes.io/projected/772d8d93-5c01-4e16-832e-51dbcf7191bd-kube-api-access-f4hnc" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964421 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d665bcd-2a5f-45c8-802a-370fc4a4aefd" volumeName="kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-daemon-config" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964433 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9425f6e8-123b-4acb-a3d1-5ccd19761401" volumeName="kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964443 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0964e1df-d8ed-402a-831f-b8996a1236f3" volumeName="kubernetes.io/empty-dir/0964e1df-d8ed-402a-831f-b8996a1236f3-tmp-dir" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964454 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0bbc3b7e-1648-4984-b4da-673a033aca22" volumeName="kubernetes.io/projected/0bbc3b7e-1648-4984-b4da-673a033aca22-kube-api-access-dtb2k" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964464 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="35ce84ec-29fd-49ab-9c7a-092e99f093b8" volumeName="kubernetes.io/projected/35ce84ec-29fd-49ab-9c7a-092e99f093b8-kube-api-access-jsx2q" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964474 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="745ded03-fed2-450a-bb70-8a78b9d3200d" volumeName="kubernetes.io/configmap/745ded03-fed2-450a-bb70-8a78b9d3200d-config" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964489 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964500 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed231fcb-17c8-4887-b2b3-1fb1c64eb442" volumeName="kubernetes.io/projected/ed231fcb-17c8-4887-b2b3-1fb1c64eb442-kube-api-access-b7w8d" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964510 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba48a24-ec24-4b58-820c-c69010288372" volumeName="kubernetes.io/projected/fba48a24-ec24-4b58-820c-c69010288372-kube-api-access-gvcfs" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964520 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0964e1df-d8ed-402a-831f-b8996a1236f3" volumeName="kubernetes.io/configmap/0964e1df-d8ed-402a-831f-b8996a1236f3-config-volume" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964531 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0964e1df-d8ed-402a-831f-b8996a1236f3" volumeName="kubernetes.io/projected/0964e1df-d8ed-402a-831f-b8996a1236f3-kube-api-access-jc8tf" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964543 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7d665bcd-2a5f-45c8-802a-370fc4a4aefd" volumeName="kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cni-binary-copy" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964553 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8505e4f1-35bd-4ff0-ad33-2febfff8fbf7" volumeName="kubernetes.io/secret/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-cert" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964564 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9425f6e8-123b-4acb-a3d1-5ccd19761401" volumeName="kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964576 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="deb16adf-e150-4c31-8b3f-6509c0b919e3" volumeName="kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-tuned" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964585 5144 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" volumeName="kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv" seLinuxMountContext="" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964595 5144 reconstruct.go:97] "Volume reconstruction finished" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.964602 5144 reconciler.go:29] "Reconciler: start to sync state" Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.965363 5144 factory.go:153] Registering CRI-O factory Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.965378 5144 factory.go:223] Registration of the crio container factory successfully Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.965430 5144 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.965449 5144 factory.go:103] Registering Raw factory Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.965464 5144 manager.go:1199] Started watching for new ooms in manager Nov 03 00:05:09.966019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.965810 5144 manager.go:319] Starting recovery of all containers Nov 03 00:05:09.983914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.983852 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:05:09.983914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.983895 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:05:09.986458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.986431 5144 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 00:05:09.987930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.987910 5144 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 00:05:09.987930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.987930 5144 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 00:05:09.988115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:09.987949 5144 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 00:05:09.988115 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:09.987990 5144 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 00:05:10.003151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.003121 5144 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="f7fd299f32d310da05fda990e7b2aff353e4a783341a71bc65ec2ca966448de2" exitCode=0 Nov 03 00:05:10.003151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.003151 5144 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="ec317e1278e616a29ca02b8f96307a717ffa61c0387b4f7afbb8ed337f244465" exitCode=0 Nov 03 00:05:10.003151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.003161 5144 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="70281a9da8f50d744623c3b321bd266a2ab35a5f3d33b1cfafc22bb73ad098e9" exitCode=0 Nov 03 00:05:10.003320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.003170 5144 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="6c017b81f0b7feb3d6510550085eb69df155e76c256d9d35c9448bd9173b599c" exitCode=0 Nov 03 00:05:10.003320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.003178 5144 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="5e41d8846367d47a914ef93051a8be115fe03b6ac3a632672a7d2be9d7aa4969" exitCode=0 Nov 03 00:05:10.003320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.003186 5144 generic.go:356] "Generic (PLEG): container finished" podID="fba48a24-ec24-4b58-820c-c69010288372" containerID="df15ea5a936fa03abb7788dce2cf1076737fd27aba9b4bb705508baac10a3e1e" exitCode=0 Nov 03 00:05:10.017680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.017656 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:05:10.017807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.017690 5144 generic.go:356] "Generic (PLEG): container finished" podID="7c5244e9-0e47-4a6d-a27d-24ce8455a537" containerID="0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435" exitCode=255 Nov 03 00:05:10.047831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.047795 5144 generic.go:356] "Generic (PLEG): container finished" podID="2a0942cf01c78617561368216297af20" containerID="c12b9ebc180ac2019d6b8125d82390dc18bd5ee28f4f81c5c86edffcabbe6c1e" exitCode=0 Nov 03 00:05:10.059791 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.059767 5144 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:05:10.062137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.062112 5144 manager.go:324] Recovery completed Nov 03 00:05:10.081842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.081817 5144 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:10.083295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.083272 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:10.083295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.083306 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:10.083430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.083316 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:10.086039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086020 5144 cpu_manager.go:226] "Starting" policy="none" Nov 03 00:05:10.086039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086037 5144 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 00:05:10.086164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086059 5144 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 00:05:10.086164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086038 5144 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:10.086222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086201 5144 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 00:05:10.086250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086217 5144 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 00:05:10.086250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086241 5144 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 00:05:10.086308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086249 5144 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 00:05:10.086308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086257 5144 policy_none.go:50] "Start" Nov 03 00:05:10.086308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086270 5144 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 00:05:10.086308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086291 5144 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 00:05:10.086446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086436 5144 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 00:05:10.086477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086447 5144 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 00:05:10.086477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.086453 5144 policy_none.go:44] "Start" Nov 03 00:05:10.087138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.087122 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:10.087215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.087156 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:10.087215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.087170 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:10.088055 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.088032 5144 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Nov 03 00:05:10.090803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.090778 5144 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 00:05:10.093938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.093923 5144 manager.go:341] "Starting Device Plugin manager" Nov 03 00:05:10.094029 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.093948 5144 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 00:05:10.094029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.093957 5144 server.go:87] "Starting device plugin registration server" Nov 03 00:05:10.094227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094214 5144 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 00:05:10.094273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094228 5144 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 00:05:10.094708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094658 5144 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 00:05:10.094801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094765 5144 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 00:05:10.094901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094813 5144 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 00:05:10.094901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094820 5144 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 00:05:10.095010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.094875 5144 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:05:10.094797427Z","UUID":"8c52a465-3bc8-48f3-a838-8574a6bf2a59","Handler":null,"Name":"","Endpoint":""} Nov 03 00:05:10.096368 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.096327 5144 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 00:05:10.096493 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.096371 5144 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-143-162.us-west-1.compute.internal\" not found" Nov 03 00:05:10.097165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.097146 5144 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 00:05:10.097246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.097176 5144 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 00:05:10.154474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.154437 5144 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:10.154669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.154541 5144 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:10.154669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.154558 5144 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:10.195312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.195291 5144 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 00:05:10.196544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.196526 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 00:05:10.196679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.196557 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 00:05:10.196679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.196571 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 00:05:10.196679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.196598 5144 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.206617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.206595 5144 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 00:05:10.206984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.206968 5144 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 00:05:10.288321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.288259 5144 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal"] Nov 03 00:05:10.288955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.288936 5144 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.289048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.288997 5144 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.289048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.289005 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"6b8cde6519a41e629e9c7fa53310b7cb","Type":"ContainerStarted","Data":"f8a9c81cc7d435376a727ff1bb5ad739b979c80da53d322f9ae82960f6ed2c17"} Nov 03 00:05:10.289190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.289048 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"6b8cde6519a41e629e9c7fa53310b7cb","Type":"ContainerStarted","Data":"e4b75b4dd6762f4d4707b7a421ed7493db8156d700ca41bcc6bbf7bef8173b55"} Nov 03 00:05:10.289190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.289083 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"2a0942cf01c78617561368216297af20","Type":"ContainerStarted","Data":"326f5e029ba1827904071effc62f2de863d27c8a0af7af7d672086c32c0f8db2"} Nov 03 00:05:10.289190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.289099 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"2a0942cf01c78617561368216297af20","Type":"ContainerDied","Data":"c12b9ebc180ac2019d6b8125d82390dc18bd5ee28f4f81c5c86edffcabbe6c1e"} Nov 03 00:05:10.289190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.289112 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" event={"ID":"2a0942cf01c78617561368216297af20","Type":"ContainerStarted","Data":"a07cff79a31c6e0db4446969e212a90c817cbd49de3dc5e2af73cd343899c4d4"} Nov 03 00:05:10.304203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.304184 5144 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.304281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.304242 5144 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.304804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.304791 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeNotReady" Nov 03 00:05:10.304904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.304822 5144 setters.go:546] "Node became not ready" node="ip-10-0-143-162.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T00:05:10Z","lastTransitionTime":"2026-11-03T00:05:10Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 00:05:10.316043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.316024 5144 nodeinfomanager.go:471] existing CSINode "ip-10-0-143-162.us-west-1.compute.internal" is owned by different node (oldNodeID="4088fb70-e5b0-4199-a322-307bd76b1b50", newNodeID=""), cleaning up... Nov 03 00:05:10.348566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.348550 5144 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:10.348712 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.348587 5144 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.348916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.348904 5144 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:10.348964 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:10.348932 5144 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.408910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.408877 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.409049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.408919 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.409049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.408943 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b8cde6519a41e629e9c7fa53310b7cb-config\") pod \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"6b8cde6519a41e629e9c7fa53310b7cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.509393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.509362 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.509393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.509394 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.509603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.509415 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b8cde6519a41e629e9c7fa53310b7cb-config\") pod \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"6b8cde6519a41e629e9c7fa53310b7cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.509603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.509467 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.509603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.509494 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2a0942cf01c78617561368216297af20-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"2a0942cf01c78617561368216297af20\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.509603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.509467 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/6b8cde6519a41e629e9c7fa53310b7cb-config\") pod \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" (UID: \"6b8cde6519a41e629e9c7fa53310b7cb\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:10.946173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:10.946133 5144 apiserver.go:52] "Watching apiserver" Nov 03 00:05:11.050882 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.050854 5144 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:11.051039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.050978 5144 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:11.099277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.099255 5144 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 00:05:11.099964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.099942 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-gtptx","openshift-ingress-canary/ingress-canary-xs77t","openshift-multus/multus-additional-cni-plugins-5mqcz","openshift-network-diagnostics/network-check-target-gbt9l","openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6","openshift-image-registry/image-registry-7c595ff78b-zkrdt","openshift-multus/multus-wbjwv","kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal","kube-system/konnectivity-agent-z8d7r","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal","openshift-insights/periodic-gathering-rj2h8-pnnjn","openshift-image-registry/image-pruner-29894400-7th8w","openshift-cluster-node-tuning-operator/tuned-lfw72","openshift-ingress/router-default-767574d6c-vss4j","openshift-network-operator/iptables-alerter-t4jtn","openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm","openshift-image-registry/node-ca-dgbsh","openshift-console-operator/console-operator-7cd56b8dbc-cdfnj","openshift-dns/node-resolver-4vplk","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w","openshift-ovn-kubernetes/ovnkube-node-8lq8f","openshift-service-ca/service-ca-5fffc97bdf-sc8zr","kube-system/kas-connection-checker-685895c7c9-mvcq4","kube-system/kas-connection-checker-685895c7c9-kzvvs","openshift-dns/dns-default-n2pqm","openshift-insights/insights-runtime-extractor-llwll","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn","openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx","kube-system/global-pull-secret-syncer-cw8kk","openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4","openshift-network-console/networking-console-plugin-86c478bb88-jm2js","openshift-image-registry/image-registry-5cd89fb64c-8bxkm","openshift-insights/insights-operator-7c4986d449-mdw87","openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6","openshift-multus/network-metrics-daemon-lcd5z"] Nov 03 00:05:11.104255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.104228 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.104397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.104304 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.106555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.106535 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.106974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.106958 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.107415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.107398 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.110708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.110689 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:11.114103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.114084 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.114733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.114717 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.115952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.115936 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.117032 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.117017 5144 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:11.118816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.118799 5144 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 00:05:11.147427 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.147401 5144 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:11.147427 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.147413 5144 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal" Nov 03 00:05:11.152845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.152828 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.153227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.153211 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.153308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.153296 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.153357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.153296 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 00:05:11.153736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.153720 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 00:05:11.153830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.153814 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.153940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.153927 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.154079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.154066 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.156850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.156831 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.157827 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.157811 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 00:05:11.157933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.157921 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 00:05:11.158382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158365 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.158435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158414 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.158509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158496 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.158601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158590 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 00:05:11.158667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158604 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.158721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158712 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 00:05:11.158769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158742 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 00:05:11.158810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158770 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.158859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158811 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 00:05:11.158859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158822 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.159013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.158990 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.159673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.159654 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 00:05:11.164681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.164661 5144 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 00:05:11.168569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.168552 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.168716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.168702 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 00:05:11.171725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.171705 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 00:05:11.171813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.171726 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 00:05:11.171813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.171746 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202219 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202226 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202248 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202282 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202307 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202294 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 00:05:11.202329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202325 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.202602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202369 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.202602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.202397 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.208724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.208701 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.208827 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.208762 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 00:05:11.208869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.208842 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 00:05:11.209568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209549 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209551 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209573 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209591 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209616 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209621 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209652 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 00:05:11.209671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209655 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209734 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"serviceca\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209740 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209771 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209806 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209813 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209826 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209829 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209860 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209919 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209921 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209966 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209965 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209999 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.209972 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210045 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210046 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210059 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210073 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210074 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210150 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210155 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210174 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210187 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 00:05:11.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210229 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 00:05:11.210787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210505 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 00:05:11.210874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.210852 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 00:05:11.211538 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.211523 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 00:05:11.212858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.212841 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-hostroot\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.212928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.212867 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54cx9\" (UniqueName: \"kubernetes.io/projected/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-kube-api-access-54cx9\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.212928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.212884 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.212928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.212899 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-kubelet\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.212928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.212914 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-etc-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.213086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.212968 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9fbr\" (UniqueName: \"kubernetes.io/projected/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-kube-api-access-x9fbr\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:05:11.213086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213053 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-stats-auth\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.213199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213183 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t5vr\" (UniqueName: \"kubernetes.io/projected/31e371fb-f718-43d0-bf0f-d909838ac2d0-kube-api-access-9t5vr\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.213313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213294 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a0eb70c-74c9-485b-a907-fa2830431a1d-konnectivity-ca\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:05:11.213373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213328 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.213373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213357 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-stats-auth\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.213473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213384 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.213473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213412 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd9lf\" (UniqueName: \"kubernetes.io/projected/88200eac-4b64-4019-b733-335740bfe1e5-kube-api-access-pd9lf\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.213473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213443 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/745ded03-fed2-450a-bb70-8a78b9d3200d-serving-cert\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:05:11.213473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213467 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213490 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213509 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-tmp\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213533 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsx2q\" (UniqueName: \"kubernetes.io/projected/35ce84ec-29fd-49ab-9c7a-092e99f093b8-kube-api-access-jsx2q\") pod \"kas-connection-checker-685895c7c9-gtptx\" (UID: \"35ce84ec-29fd-49ab-9c7a-092e99f093b8\") " pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213571 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213607 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-log-socket\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213617 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/1a0eb70c-74c9-485b-a907-fa2830431a1d-konnectivity-ca\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213656 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-system-cni-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.213669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213610 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-tmp\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213698 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213727 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/745ded03-fed2-450a-bb70-8a78b9d3200d-serving-cert\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213752 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213792 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f58b544f-c636-4c93-9e1c-9a817590d307-serving-cert\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213851 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwkv\" (UniqueName: \"kubernetes.io/projected/f58b544f-c636-4c93-9e1c-9a817590d307-kube-api-access-rpwkv\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213913 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213931 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-ovn\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213951 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-socket-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213969 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-iptables-alerter-script\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213985 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.213997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.213999 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214013 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-host\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214028 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8tz\" (UniqueName: \"kubernetes.io/projected/deb16adf-e150-4c31-8b3f-6509c0b919e3-kube-api-access-5r8tz\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214047 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214074 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-crio-socket\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214090 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214114 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6jzx\" (UniqueName: \"kubernetes.io/projected/1edd18e2-9aa6-4708-98e9-773c0a38a368-kube-api-access-j6jzx\") pod \"kas-connection-checker-685895c7c9-mvcq4\" (UID: \"1edd18e2-9aa6-4708-98e9-773c0a38a368\") " pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214129 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thvvt\" (UniqueName: \"kubernetes.io/projected/d7e0f3c7-a695-4b00-8178-1955b1159469-kube-api-access-thvvt\") pod \"kas-connection-checker-685895c7c9-kzvvs\" (UID: \"d7e0f3c7-a695-4b00-8178-1955b1159469\") " pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214160 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214233 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/f58b544f-c636-4c93-9e1c-9a817590d307-serving-cert\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214305 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214315 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214317 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-cnibin\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.214506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214486 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-iptables-alerter-script\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214538 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214668 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214703 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trkh5\" (UniqueName: \"kubernetes.io/projected/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-kube-api-access-trkh5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214762 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-lib-modules\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214796 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc8tf\" (UniqueName: \"kubernetes.io/projected/0964e1df-d8ed-402a-831f-b8996a1236f3-kube-api-access-jc8tf\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214824 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214851 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214856 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214901 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-device-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214926 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214952 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214978 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.214956 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.215126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215109 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-node-log\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215135 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-netd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215161 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-registration-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215191 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f94dde-4d2f-4969-b7ce-357282dc6529-serviceca\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215215 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-system-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215263 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-var-lib-kubelet\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215333 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215364 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovn-node-metrics-cert\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215392 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215420 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215444 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f94dde-4d2f-4969-b7ce-357282dc6529-host\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215465 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215468 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-conf-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.215725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215502 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-etc-kubernetes\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvcfs\" (UniqueName: \"kubernetes.io/projected/fba48a24-ec24-4b58-820c-c69010288372-kube-api-access-gvcfs\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215751 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215795 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-cert\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215911 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.215980 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216090 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-data-volume\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216106 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216186 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216250 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-data-volume\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216257 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-socket-dir-parent\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216303 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-kubelet\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.216333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216330 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysconfig\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216356 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216389 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wr9s\" (UniqueName: \"kubernetes.io/projected/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-api-access-9wr9s\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216395 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0f94dde-4d2f-4969-b7ce-357282dc6529-serviceca\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216412 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-key\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216450 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-cabundle\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216520 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216620 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb28b\" (UniqueName: \"kubernetes.io/projected/66a8deff-4567-470f-98d0-bec9ea590d9a-kube-api-access-hb28b\") pod \"migrator-66df5d496b-xbbfd\" (UID: \"66a8deff-4567-470f-98d0-bec9ea590d9a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216693 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-cabundle\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216800 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmqlg\" (UniqueName: \"kubernetes.io/projected/745ded03-fed2-450a-bb70-8a78b9d3200d-kube-api-access-kmqlg\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:05:11.216863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216855 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-systemd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216882 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-var-lib-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.216961 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217014 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-tmp\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217048 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217073 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-systemd\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217100 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-tmp\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217149 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.217232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217207 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bd78\" (UniqueName: \"kubernetes.io/projected/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-kube-api-access-4bd78\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217267 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217352 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4hnc\" (UniqueName: \"kubernetes.io/projected/772d8d93-5c01-4e16-832e-51dbcf7191bd-kube-api-access-f4hnc\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217419 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88200eac-4b64-4019-b733-335740bfe1e5-tmp-dir\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217447 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-env-overrides\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217473 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-k8s-cni-cncf-io\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217477 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.217517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217507 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88200eac-4b64-4019-b733-335740bfe1e5-tmp-dir\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217551 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-netns\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217597 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-default-certificate\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217669 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217687 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0964e1df-d8ed-402a-831f-b8996a1236f3-tmp-dir\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217740 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtb2k\" (UniqueName: \"kubernetes.io/projected/0bbc3b7e-1648-4984-b4da-673a033aca22-kube-api-access-dtb2k\") pod \"volume-data-source-validator-58c9f5f46d-lnvpp\" (UID: \"0bbc3b7e-1648-4984-b4da-673a033aca22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217747 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0964e1df-d8ed-402a-831f-b8996a1236f3-tmp-dir\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217778 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-service-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.217831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217807 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-env-overrides\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217842 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217875 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217885 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-default-certificate\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217937 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-multus-certs\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217948 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f58b544f-c636-4c93-9e1c-9a817590d307-service-ca-bundle\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.217969 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218002 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.218120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218025 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-daemon-config\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218131 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-kubelet-config\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218166 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218179 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-daemon-config\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218191 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-trusted-ca\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218223 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfhs5\" (UniqueName: \"kubernetes.io/projected/82a02683-66a4-43a5-9d32-c53f72d1f613-kube-api-access-vfhs5\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218265 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cni-binary-copy\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218317 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-multus\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218356 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-netns\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218399 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-config\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218434 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-sys-fs\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218451 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218462 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218486 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-config\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.218498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218491 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cni-binary-copy\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218510 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218538 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218562 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218582 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218592 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-trusted-ca\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218602 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-config\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218597 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-bin\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218670 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218699 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218722 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-dbus\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218735 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7c5244e9-0e47-4a6d-a27d-24ce8455a537-config\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218746 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218781 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-systemd-units\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218831 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-447n8\" (UniqueName: \"kubernetes.io/projected/7c5244e9-0e47-4a6d-a27d-24ce8455a537-kube-api-access-447n8\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218875 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-snapshots\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218879 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.218982 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-modprobe-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219012 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-sys\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219030 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/f58b544f-c636-4c93-9e1c-9a817590d307-snapshots\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219037 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88200eac-4b64-4019-b733-335740bfe1e5-hosts-file\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219088 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-bin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219121 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qj2\" (UniqueName: \"kubernetes.io/projected/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-kube-api-access-n4qj2\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.219205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219218 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219251 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219263 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219291 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a0eb70c-74c9-485b-a907-fa2830431a1d-agent-certs\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219314 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219331 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cnibin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219355 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-kubernetes\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219377 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-run\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219412 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c5244e9-0e47-4a6d-a27d-24ce8455a537-serving-cert\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219455 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219459 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw657\" (UniqueName: \"kubernetes.io/projected/f0f94dde-4d2f-4969-b7ce-357282dc6529-kube-api-access-kw657\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219500 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae7af53a-1838-409e-b5f3-6aa5c688dd55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219528 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-conf\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219552 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-tuned\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219577 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgq4q\" (UniqueName: \"kubernetes.io/projected/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-kube-api-access-dgq4q\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219610 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219671 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7w8d\" (UniqueName: \"kubernetes.io/projected/ed231fcb-17c8-4887-b2b3-1fb1c64eb442-kube-api-access-b7w8d\") pod \"migrator-66df5d496b-zqsm4\" (UID: \"ed231fcb-17c8-4887-b2b3-1fb1c64eb442\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219685 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-tuned\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219696 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-script-lib\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219703 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219731 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd7n4\" (UniqueName: \"kubernetes.io/projected/e8fc0eda-47b5-49fb-86bd-3e23f563c46b-kube-api-access-bd7n4\") pod \"network-check-source-6f78d59f99-dlhnx\" (UID: \"e8fc0eda-47b5-49fb-86bd-3e23f563c46b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219762 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-779m4\" (UniqueName: \"kubernetes.io/projected/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-kube-api-access-779m4\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219773 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/ae7af53a-1838-409e-b5f3-6aa5c688dd55-nginx-conf\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219789 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219815 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/745ded03-fed2-450a-bb70-8a78b9d3200d-config\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219840 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-slash\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219865 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjgg\" (UniqueName: \"kubernetes.io/projected/ca0220e0-47de-45fb-bbb6-11ed27994448-kube-api-access-wgjgg\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219870 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovnkube-script-lib\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219875 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7c5244e9-0e47-4a6d-a27d-24ce8455a537-serving-cert\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219890 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-host-slash\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219914 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0964e1df-d8ed-402a-831f-b8996a1236f3-config-volume\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.219949 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.220023 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/745ded03-fed2-450a-bb70-8a78b9d3200d-config\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.220078 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0964e1df-d8ed-402a-831f-b8996a1236f3-config-volume\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.220212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.220142 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fba48a24-ec24-4b58-820c-c69010288372-cni-binary-copy\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.255425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.255406 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 00:05:11.255585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.255570 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-rdw2q\"" Nov 03 00:05:11.255787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.255773 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 00:05:11.255855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.255825 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 00:05:11.255855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.255843 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Nov 03 00:05:11.255988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.255929 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-nhzpn\"" Nov 03 00:05:11.256111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256068 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 00:05:11.256111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256077 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-scmmg\"" Nov 03 00:05:11.256111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256075 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-vqr8l\"" Nov 03 00:05:11.256111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256102 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-cert\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:05:11.256303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256140 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 00:05:11.256303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256252 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-q5tjj\"" Nov 03 00:05:11.256388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256343 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.256388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256368 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-klhlc\"" Nov 03 00:05:11.256464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256406 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.256509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256487 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 00:05:11.256769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256566 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"pruner-dockercfg-p464f\"" Nov 03 00:05:11.256769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256574 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 00:05:11.256769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256709 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256799 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256819 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-jkmg8\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256832 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k6628\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256860 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-52krg\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256863 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-f67qw\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256871 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-kf6rl\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256928 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-gd4nl\"" Nov 03 00:05:11.256952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256939 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-signing-key\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:11.257217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256972 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-fmpf5\"" Nov 03 00:05:11.257217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.256989 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-pkcxp\"" Nov 03 00:05:11.257217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257038 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-g7949\"" Nov 03 00:05:11.257217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257083 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-ptgp6\"" Nov 03 00:05:11.257217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257137 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 00:05:11.257421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257295 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bkhng\"" Nov 03 00:05:11.257421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257318 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8glgl\"" Nov 03 00:05:11.257421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257344 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-gj4lf\"" Nov 03 00:05:11.257421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257345 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z6zhf\"" Nov 03 00:05:11.257421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.257369 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 00:05:11.259564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.259544 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2ee7c64c-4945-469a-8d27-cfb702aabf50-original-pull-secret\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.259644 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.259548 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/1a0eb70c-74c9-485b-a907-fa2830431a1d-agent-certs\") pod \"konnectivity-agent-z8d7r\" (UID: \"1a0eb70c-74c9-485b-a907-fa2830431a1d\") " pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:05:11.259644 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.259589 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.264935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.264913 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.266400 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.266377 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-ovn-node-metrics-cert\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.275099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.275071 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd9lf\" (UniqueName: \"kubernetes.io/projected/88200eac-4b64-4019-b733-335740bfe1e5-kube-api-access-pd9lf\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.275758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.275737 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54cx9\" (UniqueName: \"kubernetes.io/projected/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-kube-api-access-54cx9\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.275978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.275957 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") pod \"periodic-gathering-rj2h8-pnnjn\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:05:11.276044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.276023 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.276126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.276108 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9fbr\" (UniqueName: \"kubernetes.io/projected/8505e4f1-35bd-4ff0-ad33-2febfff8fbf7-kube-api-access-x9fbr\") pod \"ingress-canary-xs77t\" (UID: \"8505e4f1-35bd-4ff0-ad33-2febfff8fbf7\") " pod="openshift-ingress-canary/ingress-canary-xs77t" Nov 03 00:05:11.276246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.276229 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwkv\" (UniqueName: \"kubernetes.io/projected/f58b544f-c636-4c93-9e1c-9a817590d307-kube-api-access-rpwkv\") pod \"insights-operator-7c4986d449-mdw87\" (UID: \"f58b544f-c636-4c93-9e1c-9a817590d307\") " pod="openshift-insights/insights-operator-7c4986d449-mdw87" Nov 03 00:05:11.276653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.276618 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trkh5\" (UniqueName: \"kubernetes.io/projected/33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59-kube-api-access-trkh5\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w\" (UID: \"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" Nov 03 00:05:11.277151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.277132 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.277221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.277202 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6jzx\" (UniqueName: \"kubernetes.io/projected/1edd18e2-9aa6-4708-98e9-773c0a38a368-kube-api-access-j6jzx\") pod \"kas-connection-checker-685895c7c9-mvcq4\" (UID: \"1edd18e2-9aa6-4708-98e9-773c0a38a368\") " pod="kube-system/kas-connection-checker-685895c7c9-mvcq4" Nov 03 00:05:11.277606 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.277588 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc8tf\" (UniqueName: \"kubernetes.io/projected/0964e1df-d8ed-402a-831f-b8996a1236f3-kube-api-access-jc8tf\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.277606 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.277596 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t5vr\" (UniqueName: \"kubernetes.io/projected/31e371fb-f718-43d0-bf0f-d909838ac2d0-kube-api-access-9t5vr\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.277785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.277771 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsx2q\" (UniqueName: \"kubernetes.io/projected/35ce84ec-29fd-49ab-9c7a-092e99f093b8-kube-api-access-jsx2q\") pod \"kas-connection-checker-685895c7c9-gtptx\" (UID: \"35ce84ec-29fd-49ab-9c7a-092e99f093b8\") " pod="kube-system/kas-connection-checker-685895c7c9-gtptx" Nov 03 00:05:11.280958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.280938 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.281191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.281173 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thvvt\" (UniqueName: \"kubernetes.io/projected/d7e0f3c7-a695-4b00-8178-1955b1159469-kube-api-access-thvvt\") pod \"kas-connection-checker-685895c7c9-kzvvs\" (UID: \"d7e0f3c7-a695-4b00-8178-1955b1159469\") " pod="kube-system/kas-connection-checker-685895c7c9-kzvvs" Nov 03 00:05:11.281314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.281295 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8tz\" (UniqueName: \"kubernetes.io/projected/deb16adf-e150-4c31-8b3f-6509c0b919e3-kube-api-access-5r8tz\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.281365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.281352 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4hnc\" (UniqueName: \"kubernetes.io/projected/772d8d93-5c01-4e16-832e-51dbcf7191bd-kube-api-access-f4hnc\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:11.281466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.281451 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmqm4\" (UniqueName: \"kubernetes.io/projected/6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1-kube-api-access-qmqm4\") pod \"network-check-target-gbt9l\" (UID: \"6f87535b-ae3b-45e3-abd6-ca3ca4b9bdb1\") " pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:11.282832 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.282815 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wr9s\" (UniqueName: \"kubernetes.io/projected/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-api-access-9wr9s\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.285090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.285075 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286543 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-779m4\" (UniqueName: \"kubernetes.io/projected/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-kube-api-access-779m4\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286719 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7w8d\" (UniqueName: \"kubernetes.io/projected/ed231fcb-17c8-4887-b2b3-1fb1c64eb442-kube-api-access-b7w8d\") pod \"migrator-66df5d496b-zqsm4\" (UID: \"ed231fcb-17c8-4887-b2b3-1fb1c64eb442\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-zqsm4" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286750 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb28b\" (UniqueName: \"kubernetes.io/projected/66a8deff-4567-470f-98d0-bec9ea590d9a-kube-api-access-hb28b\") pod \"migrator-66df5d496b-xbbfd\" (UID: \"66a8deff-4567-470f-98d0-bec9ea590d9a\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-xbbfd" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286757 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjgg\" (UniqueName: \"kubernetes.io/projected/ca0220e0-47de-45fb-bbb6-11ed27994448-kube-api-access-wgjgg\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286809 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bd78\" (UniqueName: \"kubernetes.io/projected/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-kube-api-access-4bd78\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286929 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvcfs\" (UniqueName: \"kubernetes.io/projected/fba48a24-ec24-4b58-820c-c69010288372-kube-api-access-gvcfs\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.286996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.286964 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfhs5\" (UniqueName: \"kubernetes.io/projected/82a02683-66a4-43a5-9d32-c53f72d1f613-kube-api-access-vfhs5\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.287258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.287106 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgq4q\" (UniqueName: \"kubernetes.io/projected/f29e4053-bef9-482f-8fe1-44a0d9bd4dfd-kube-api-access-dgq4q\") pod \"service-ca-5fffc97bdf-sc8zr\" (UID: \"f29e4053-bef9-482f-8fe1-44a0d9bd4dfd\") " pod="openshift-service-ca/service-ca-5fffc97bdf-sc8zr" Nov 03 00:05:11.287365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.287343 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmqlg\" (UniqueName: \"kubernetes.io/projected/745ded03-fed2-450a-bb70-8a78b9d3200d-kube-api-access-kmqlg\") pod \"service-ca-operator-544d749c8c-xxhj6\" (UID: \"745ded03-fed2-450a-bb70-8a78b9d3200d\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" Nov 03 00:05:11.287729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.287712 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw657\" (UniqueName: \"kubernetes.io/projected/f0f94dde-4d2f-4969-b7ce-357282dc6529-kube-api-access-kw657\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.288025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.288008 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtb2k\" (UniqueName: \"kubernetes.io/projected/0bbc3b7e-1648-4984-b4da-673a033aca22-kube-api-access-dtb2k\") pod \"volume-data-source-validator-58c9f5f46d-lnvpp\" (UID: \"0bbc3b7e-1648-4984-b4da-673a033aca22\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-lnvpp" Nov 03 00:05:11.288386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.288364 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd7n4\" (UniqueName: \"kubernetes.io/projected/e8fc0eda-47b5-49fb-86bd-3e23f563c46b-kube-api-access-bd7n4\") pod \"network-check-source-6f78d59f99-dlhnx\" (UID: \"e8fc0eda-47b5-49fb-86bd-3e23f563c46b\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-dlhnx" Nov 03 00:05:11.288745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.288731 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") pod \"image-pruner-29894400-7th8w\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:05:11.288910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.288890 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-447n8\" (UniqueName: \"kubernetes.io/projected/7c5244e9-0e47-4a6d-a27d-24ce8455a537-kube-api-access-447n8\") pod \"console-operator-7cd56b8dbc-cdfnj\" (UID: \"7c5244e9-0e47-4a6d-a27d-24ce8455a537\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:11.289700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.289681 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qj2\" (UniqueName: \"kubernetes.io/projected/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-kube-api-access-n4qj2\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.320468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320447 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-socket-dir-parent\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320479 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-kubelet\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320502 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysconfig\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.320581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320562 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysconfig\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.320581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320562 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320597 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-systemd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320611 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-var-lib-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320625 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-systemd\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-socket-dir-parent\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320661 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320668 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-systemd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-kubelet\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320699 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-systemd\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320705 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-k8s-cni-cncf-io\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320722 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-var-lib-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320731 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-netns\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320770 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-k8s-cni-cncf-io\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320777 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-netns\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.320788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320789 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320815 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-multus-certs\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320837 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320839 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320862 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320869 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-run-multus-certs\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320886 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-kubelet-config\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320913 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-multus\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320934 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-netns\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.320946 5144 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.320946 5144 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.320959 5144 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.320964 5144 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320977 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-kubelet-config\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320986 5144 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321004 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-sys-fs\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.321029 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.821011069 +0000 UTC m=+1.966705520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321037 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-netns\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.320957 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-sys-fs\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.321049 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.821039575 +0000 UTC m=+1.966734009 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321039 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-multus\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321080 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321097 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321104 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-bin\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321133 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-dbus\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321147 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321154 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-systemd-units\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321163 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-bin\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321195 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-systemd-units\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321200 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2ee7c64c-4945-469a-8d27-cfb702aabf50-dbus\") pod \"global-pull-secret-syncer-cw8kk\" (UID: \"2ee7c64c-4945-469a-8d27-cfb702aabf50\") " pod="kube-system/global-pull-secret-syncer-cw8kk" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321226 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-modprobe-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321246 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-sys\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321268 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88200eac-4b64-4019-b733-335740bfe1e5-hosts-file\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321289 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-bin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321311 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321313 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/88200eac-4b64-4019-b733-335740bfe1e5-hosts-file\") pod \"node-resolver-4vplk\" (UID: \"88200eac-4b64-4019-b733-335740bfe1e5\") " pod="openshift-dns/node-resolver-4vplk" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321319 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-sys\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321334 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cnibin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321349 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-modprobe-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321357 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-kubernetes\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321377 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-run\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321400 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-conf\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321406 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-cnibin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321427 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-slash\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321432 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-kubernetes\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321444 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-host-var-lib-cni-bin\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321450 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-run\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321451 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-host-slash\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321490 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-slash\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321492 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321548 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-conf\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321583 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-etc-sysctl-d\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321584 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c44bcac-ee48-48af-b5b4-99f2ce0e81d6-host-slash\") pod \"iptables-alerter-t4jtn\" (UID: \"2c44bcac-ee48-48af-b5b4-99f2ce0e81d6\") " pod="openshift-network-operator/iptables-alerter-t4jtn" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321655 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-hostroot\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.321649 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.321666 5144 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321684 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-kubelet\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321706 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-etc-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321711 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-hostroot\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.321727 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.821709172 +0000 UTC m=+1.967403621 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321739 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-etc-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321761 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-kubelet\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321761 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321785 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321805 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-log-socket\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321830 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-system-cni-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321838 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-log-socket\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321857 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321874 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-system-cni-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321883 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321906 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-run-ovn-kubernetes\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321910 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321931 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321935 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-ovn\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321962 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-socket-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321963 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-ovn\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.321993 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-host\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322013 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-socket-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322016 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322016 5144 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322057 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-host\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322065 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.822049663 +0000 UTC m=+1.967744112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : secret "metrics-daemon-secret" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322082 5144 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322094 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-crio-socket\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322123 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.822109569 +0000 UTC m=+1.967804031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322140 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322150 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-crio-socket\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322163 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-cnibin\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322184 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322196 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-etc-selinux\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322210 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-lib-modules\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322229 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fba48a24-ec24-4b58-820c-c69010288372-cnibin\") pod \"multus-additional-cni-plugins-5mqcz\" (UID: \"fba48a24-ec24-4b58-820c-c69010288372\") " pod="openshift-multus/multus-additional-cni-plugins-5mqcz" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322233 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-device-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322241 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-run-openvswitch\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322258 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322290 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-device-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322291 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322316 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322339 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-node-log\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322361 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-netd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322381 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.822368409 +0000 UTC m=+1.968062863 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322406 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-registration-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322407 5144 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322400 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-host-cni-netd\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322451 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.822441497 +0000 UTC m=+1.968135969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322460 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ca0220e0-47de-45fb-bbb6-11ed27994448-registration-dir\") pod \"aws-ebs-csi-driver-node-tgjtn\" (UID: \"ca0220e0-47de-45fb-bbb6-11ed27994448\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tgjtn" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322488 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4-node-log\") pod \"ovnkube-node-8lq8f\" (UID: \"c4f3669a-ce7d-4df7-a9d6-7da101fe3fd4\") " pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322519 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-system-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322545 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-var-lib-kubelet\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322583 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322608 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f94dde-4d2f-4969-b7ce-357282dc6529-host\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322669 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-var-lib-kubelet\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322675 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-system-cni-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322679 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-conf-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322707 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0f94dde-4d2f-4969-b7ce-357282dc6529-host\") pod \"node-ca-dgbsh\" (UID: \"f0f94dde-4d2f-4969-b7ce-357282dc6529\") " pod="openshift-image-registry/node-ca-dgbsh" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322714 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-etc-kubernetes\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322740 5144 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322743 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-etc-kubernetes\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322765 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/deb16adf-e150-4c31-8b3f-6509c0b919e3-lib-modules\") pod \"tuned-lfw72\" (UID: \"deb16adf-e150-4c31-8b3f-6509c0b919e3\") " pod="openshift-cluster-node-tuning-operator/tuned-lfw72" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.322714 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7d665bcd-2a5f-45c8-802a-370fc4a4aefd-multus-conf-dir\") pod \"multus-wbjwv\" (UID: \"7d665bcd-2a5f-45c8-802a-370fc4a4aefd\") " pod="openshift-multus/multus-wbjwv" Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.322830 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:11.822802709 +0000 UTC m=+1.968497158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:05:11.323566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.323447 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0964e1df-d8ed-402a-831f-b8996a1236f3-metrics-tls\") pod \"dns-default-n2pqm\" (UID: \"0964e1df-d8ed-402a-831f-b8996a1236f3\") " pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.325427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.324767 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/ae7af53a-1838-409e-b5f3-6aa5c688dd55-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-jm2js\" (UID: \"ae7af53a-1838-409e-b5f3-6aa5c688dd55\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.406432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.406407 5144 scope.go:122] "RemoveContainer" containerID="0f896c6d4ecdb02f7cd37d5e74993b33afb0e8b9afb0697d965fa7deb98a4435" Nov 03 00:05:11.408450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.408431 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" Nov 03 00:05:11.415302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.415283 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:11.468761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.468734 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:11.796556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.796525 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-jm2js"] Nov 03 00:05:11.799817 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:11.799790 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae7af53a_1838_409e_b5f3_6aa5c688dd55.slice/crio-aee549f69ff20ad3a5bd9dcf41c6e1d630a9b9b1c42377ac980edd9191e9ff10 WatchSource:0}: Error finding container aee549f69ff20ad3a5bd9dcf41c6e1d630a9b9b1c42377ac980edd9191e9ff10: Status 404 returned error can't find the container with id aee549f69ff20ad3a5bd9dcf41c6e1d630a9b9b1c42377ac980edd9191e9ff10 Nov 03 00:05:11.802225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.802207 5144 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:05:11.827709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.827683 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:11.827801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.827722 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:11.827801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.827742 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:11.827871 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827820 5144 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:05:11.827871 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827839 5144 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:11.827871 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827864 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.827847796 +0000 UTC m=+2.973542244 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : secret "metrics-daemon-secret" not found Nov 03 00:05:11.827871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.827859 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827894 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.827878278 +0000 UTC m=+2.973572716 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827906 5144 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.827915 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827928 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.827916476 +0000 UTC m=+2.973610911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827947 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.827941101 +0000 UTC m=+2.973635535 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.827960 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.827970 5144 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:11.828008 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828003 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.827991777 +0000 UTC m=+2.973686217 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828017 5144 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.828031 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828038 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.828030787 +0000 UTC m=+2.973725222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : secret "router-metrics-certs-default" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828075 5144 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.828073 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828083 5144 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cd89fb64c-8bxkm: secret "image-registry-tls" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828101 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls podName:e7ac0dbb-01b6-4660-abca-8b8c2a7d741a nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.828096154 +0000 UTC m=+2.973790588 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls") pod "image-registry-5cd89fb64c-8bxkm" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a") : secret "image-registry-tls" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828142 5144 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828151 5144 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c595ff78b-zkrdt: secret "image-registry-tls" not found Nov 03 00:05:11.828266 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:11.828200 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls podName:df250e4d-45bd-4a42-9faa-a2ed51f7589b nodeName:}" failed. No retries permitted until 2026-11-03 00:05:12.828175433 +0000 UTC m=+2.973869870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls") pod "image-registry-7c595ff78b-zkrdt" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b") : secret "image-registry-tls" not found Nov 03 00:05:11.847674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.847653 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-n2pqm"] Nov 03 00:05:11.850573 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:11.850553 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0964e1df_d8ed_402a_831f_b8996a1236f3.slice/crio-6704d7d1aa573c431ef540e080339d65691cd26b2ac2d4a78abb6b4b8bed9d9f WatchSource:0}: Error finding container 6704d7d1aa573c431ef540e080339d65691cd26b2ac2d4a78abb6b4b8bed9d9f: Status 404 returned error can't find the container with id 6704d7d1aa573c431ef540e080339d65691cd26b2ac2d4a78abb6b4b8bed9d9f Nov 03 00:05:11.933796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:11.933771 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:12.056153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.056087 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:05:12.056593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.056188 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" event={"ID":"7c5244e9-0e47-4a6d-a27d-24ce8455a537","Type":"ContainerStarted","Data":"7dd83f7e18f1dfba05d3caabccb123655f7d23ca564a7305a08ab068a65e9d09"} Nov 03 00:05:12.057471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.057446 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" event={"ID":"ae7af53a-1838-409e-b5f3-6aa5c688dd55","Type":"ContainerStarted","Data":"aee549f69ff20ad3a5bd9dcf41c6e1d630a9b9b1c42377ac980edd9191e9ff10"} Nov 03 00:05:12.058690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.058663 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2pqm" event={"ID":"0964e1df-d8ed-402a-831f-b8996a1236f3","Type":"ContainerStarted","Data":"6704d7d1aa573c431ef540e080339d65691cd26b2ac2d4a78abb6b4b8bed9d9f"} Nov 03 00:05:12.838281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838249 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:12.838458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838302 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:12.838458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838339 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:12.838458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838365 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:12.838458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838389 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:12.838458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838443 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:12.838749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838466 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:12.838749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.838528 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:12.838749 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.838671 5144 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:12.838749 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.838729 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.838712017 +0000 UTC m=+4.984406455 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:12.839055 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839038 5144 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:05:12.839144 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839093 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.839078397 +0000 UTC m=+4.984772832 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : secret "metrics-daemon-secret" not found Nov 03 00:05:12.839204 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839149 5144 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:05:12.839204 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839185 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.839173807 +0000 UTC m=+4.984868247 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:05:12.839317 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839248 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.839236805 +0000 UTC m=+4.984931242 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:05:12.839317 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839299 5144 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:12.839414 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:12.839330 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:14.839320553 +0000 UTC m=+4.985014991 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:12.842801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.842728 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"image-registry-7c595ff78b-zkrdt\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:12.842801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.842746 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31e371fb-f718-43d0-bf0f-d909838ac2d0-metrics-certs\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:12.842801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.842751 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"image-registry-5cd89fb64c-8bxkm\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:12.908166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.908090 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:12.915227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:12.915204 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:13.066749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:13.066711 5144 generic.go:356] "Generic (PLEG): container finished" podID="f58b544f-c636-4c93-9e1c-9a817590d307" containerID="11cbea1dfb216d6eef0305df8d92fb9e5ab9cf85c29a209fc0a9be03aaba9fa2" exitCode=0 Nov 03 00:05:13.067184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:13.066908 5144 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:13.068072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:13.067308 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mdw87" event={"ID":"f58b544f-c636-4c93-9e1c-9a817590d307","Type":"ContainerDied","Data":"11cbea1dfb216d6eef0305df8d92fb9e5ab9cf85c29a209fc0a9be03aaba9fa2"} Nov 03 00:05:13.068072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:13.067693 5144 scope.go:122] "RemoveContainer" containerID="11cbea1dfb216d6eef0305df8d92fb9e5ab9cf85c29a209fc0a9be03aaba9fa2" Nov 03 00:05:13.346454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:13.346415 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cd89fb64c-8bxkm"] Nov 03 00:05:13.348663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:13.348622 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c595ff78b-zkrdt"] Nov 03 00:05:13.349779 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:13.349750 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7ac0dbb_01b6_4660_abca_8b8c2a7d741a.slice/crio-d614b04b922a9a43db2eb85d6c52e178c397d65793703f74844031e199b9c90d WatchSource:0}: Error finding container d614b04b922a9a43db2eb85d6c52e178c397d65793703f74844031e199b9c90d: Status 404 returned error can't find the container with id d614b04b922a9a43db2eb85d6c52e178c397d65793703f74844031e199b9c90d Nov 03 00:05:13.391206 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:13.391172 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf250e4d_45bd_4a42_9faa_a2ed51f7589b.slice/crio-f7741e30ed6aa165676fa6a2fad10d851482dd12891fa84eb389b33e1e42c5aa WatchSource:0}: Error finding container f7741e30ed6aa165676fa6a2fad10d851482dd12891fa84eb389b33e1e42c5aa: Status 404 returned error can't find the container with id f7741e30ed6aa165676fa6a2fad10d851482dd12891fa84eb389b33e1e42c5aa Nov 03 00:05:14.071648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.071590 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" event={"ID":"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a","Type":"ContainerStarted","Data":"12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b"} Nov 03 00:05:14.072063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.071678 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" event={"ID":"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a","Type":"ContainerStarted","Data":"d614b04b922a9a43db2eb85d6c52e178c397d65793703f74844031e199b9c90d"} Nov 03 00:05:14.073086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.073061 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-jm2js" event={"ID":"ae7af53a-1838-409e-b5f3-6aa5c688dd55","Type":"ContainerStarted","Data":"79ed10818fc90af5f525ed839a77feb5d67d269176114c4db187d941b00701a7"} Nov 03 00:05:14.074622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.074593 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" event={"ID":"df250e4d-45bd-4a42-9faa-a2ed51f7589b","Type":"ContainerStarted","Data":"42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5"} Nov 03 00:05:14.074738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.074661 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" event={"ID":"df250e4d-45bd-4a42-9faa-a2ed51f7589b","Type":"ContainerStarted","Data":"f7741e30ed6aa165676fa6a2fad10d851482dd12891fa84eb389b33e1e42c5aa"} Nov 03 00:05:14.076906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.076886 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mdw87" event={"ID":"f58b544f-c636-4c93-9e1c-9a817590d307","Type":"ContainerStarted","Data":"4a10bd910d35780899a2e0ffbeaedc4fc48df054c2d95f33913e36139c93d4e4"} Nov 03 00:05:14.865370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.865288 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:14.865370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.865343 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.865374 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.865407 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865431 5144 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865486 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.865472068 +0000 UTC m=+9.011166502 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:14.865432 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865494 5144 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865503 5144 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865520 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle podName:31e371fb-f718-43d0-bf0f-d909838ac2d0 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.865507116 +0000 UTC m=+9.011201551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle") pod "router-default-767574d6c-vss4j" (UID: "31e371fb-f718-43d0-bf0f-d909838ac2d0") : configmap references non-existent config key: service-ca.crt Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865541 5144 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865551 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs podName:772d8d93-5c01-4e16-832e-51dbcf7191bd nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.865544252 +0000 UTC m=+9.011238686 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs") pod "network-metrics-daemon-lcd5z" (UID: "772d8d93-5c01-4e16-832e-51dbcf7191bd") : secret "metrics-daemon-secret" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865562 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls podName:82a02683-66a4-43a5-9d32-c53f72d1f613 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.865557082 +0000 UTC m=+9.011251516 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-b48t6" (UID: "82a02683-66a4-43a5-9d32-c53f72d1f613") : secret "samples-operator-tls" not found Nov 03 00:05:14.865577 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:14.865585 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls podName:19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:18.865569509 +0000 UTC m=+9.011263944 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-mbxcm" (UID: "19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d") : secret "cluster-monitoring-operator-tls" not found Nov 03 00:05:15.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.087919 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2pqm" event={"ID":"0964e1df-d8ed-402a-831f-b8996a1236f3","Type":"ContainerStarted","Data":"d6353cd962dda85ede8beda62282f9dccdc7ba100713cc557aea423307e80c02"} Nov 03 00:05:15.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.087952 5144 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:15.088410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.088017 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-n2pqm" event={"ID":"0964e1df-d8ed-402a-831f-b8996a1236f3","Type":"ContainerStarted","Data":"9d4084826de0c46f5cb77509e0ebc2ffa3ba99fcf62c1702b435e4405fb3ce9c"} Nov 03 00:05:15.088410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.088211 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:15.088545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.088528 5144 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:15.615509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.615476 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:15.682910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:15.682882 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:17.101178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:17.101143 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-z8d7r" Nov 03 00:05:18.405706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.405672 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:18.906332 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.906294 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:18.906522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.906345 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:18.906522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.906367 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:18.906522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.906389 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:18.906522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.906405 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:18.906522 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:18.906452 5144 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 00:05:18.906828 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:18.906533 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls podName:1cc9f90a-15a5-41ec-9ad8-90969c74c8d4 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:26.906513153 +0000 UTC m=+17.052207592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls") pod "insights-runtime-extractor-llwll" (UID: "1cc9f90a-15a5-41ec-9ad8-90969c74c8d4") : secret "insights-runtime-extractor-tls" not found Nov 03 00:05:18.907203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.907124 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/31e371fb-f718-43d0-bf0f-d909838ac2d0-service-ca-bundle\") pod \"router-default-767574d6c-vss4j\" (UID: \"31e371fb-f718-43d0-bf0f-d909838ac2d0\") " pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:18.909073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.909051 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/772d8d93-5c01-4e16-832e-51dbcf7191bd-metrics-certs\") pod \"network-metrics-daemon-lcd5z\" (UID: \"772d8d93-5c01-4e16-832e-51dbcf7191bd\") " pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:18.909140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.909084 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-mbxcm\" (UID: \"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:18.909184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.909160 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/82a02683-66a4-43a5-9d32-c53f72d1f613-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-b48t6\" (UID: \"82a02683-66a4-43a5-9d32-c53f72d1f613\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:18.911936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:18.911915 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-lcd5z" Nov 03 00:05:19.205453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.205371 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" Nov 03 00:05:19.205602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.205472 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" Nov 03 00:05:19.207041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.207027 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:19.284166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.284073 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-lcd5z"] Nov 03 00:05:19.589694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.589474 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-767574d6c-vss4j"] Nov 03 00:05:19.592125 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:19.592098 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31e371fb_f718_43d0_bf0f_d909838ac2d0.slice/crio-a75a96ed73a8aaf69264ad7776ef9082a6dd6122e45c8f67ee1ae2d443979cf2 WatchSource:0}: Error finding container a75a96ed73a8aaf69264ad7776ef9082a6dd6122e45c8f67ee1ae2d443979cf2: Status 404 returned error can't find the container with id a75a96ed73a8aaf69264ad7776ef9082a6dd6122e45c8f67ee1ae2d443979cf2 Nov 03 00:05:19.639049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.639007 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6"] Nov 03 00:05:19.639765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:19.639689 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm"] Nov 03 00:05:19.644772 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:19.642768 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19f422bb_0c3f_4b3b_b8bd_72ef8d0e655d.slice/crio-416175a90ee545e781b2dd42c4cfa92c8d883199724a735eeefe3d49f0a24ef5 WatchSource:0}: Error finding container 416175a90ee545e781b2dd42c4cfa92c8d883199724a735eeefe3d49f0a24ef5: Status 404 returned error can't find the container with id 416175a90ee545e781b2dd42c4cfa92c8d883199724a735eeefe3d49f0a24ef5 Nov 03 00:05:20.107223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.107187 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcd5z" event={"ID":"772d8d93-5c01-4e16-832e-51dbcf7191bd","Type":"ContainerStarted","Data":"f9cc01497ef0d02cf8b1c3093b951e5cb0b9c56b9c2780cba62bcb71e9e7f0d0"} Nov 03 00:05:20.108424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.108391 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" event={"ID":"82a02683-66a4-43a5-9d32-c53f72d1f613","Type":"ContainerStarted","Data":"340e36f05586125c39df586f345d07073ad7714baf51d95e45b731d1a2fd714d"} Nov 03 00:05:20.110001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.109974 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-767574d6c-vss4j" event={"ID":"31e371fb-f718-43d0-bf0f-d909838ac2d0","Type":"ContainerStarted","Data":"5a6ef34e848111887554ef233cb14b63e706b1a6f7d05e531eaa540d743d5a6a"} Nov 03 00:05:20.110125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.110014 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-767574d6c-vss4j" event={"ID":"31e371fb-f718-43d0-bf0f-d909838ac2d0","Type":"ContainerStarted","Data":"a75a96ed73a8aaf69264ad7776ef9082a6dd6122e45c8f67ee1ae2d443979cf2"} Nov 03 00:05:20.111584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.111557 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" event={"ID":"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d","Type":"ContainerStarted","Data":"416175a90ee545e781b2dd42c4cfa92c8d883199724a735eeefe3d49f0a24ef5"} Nov 03 00:05:20.208033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.207998 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:20.210542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.210519 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:20.469194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.469116 5144 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-143-162.us-west-1.compute.internal" event="NodeReady" Nov 03 00:05:20.614139 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.614110 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:20.614619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.614245 5144 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 00:05:20.619102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:20.619080 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-cdfnj" Nov 03 00:05:21.124533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:21.124443 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcd5z" event={"ID":"772d8d93-5c01-4e16-832e-51dbcf7191bd","Type":"ContainerStarted","Data":"75991c5b476057ceea4c06a7b19447dd2cefb474cf3a60dd1bce6ef510f97ba7"} Nov 03 00:05:21.124533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:21.124529 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-lcd5z" event={"ID":"772d8d93-5c01-4e16-832e-51dbcf7191bd","Type":"ContainerStarted","Data":"3561fad95b6af905e3f35657fbbc6bef0cafe176b9f39459245f1464db95d694"} Nov 03 00:05:21.125121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:21.125097 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:21.126419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:21.126397 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-767574d6c-vss4j" Nov 03 00:05:22.132435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:22.132292 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" event={"ID":"82a02683-66a4-43a5-9d32-c53f72d1f613","Type":"ContainerStarted","Data":"3f99100d201d9a291b35965d50879b0992af388bb7c7e5823c925c1adc4105e0"} Nov 03 00:05:22.132435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:22.132358 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-b48t6" event={"ID":"82a02683-66a4-43a5-9d32-c53f72d1f613","Type":"ContainerStarted","Data":"68c9869e5ce564ba5c1b34b8990dc0b2c6b0bf9550eb40fcbeece2937d4d236c"} Nov 03 00:05:23.138768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:23.138734 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-mbxcm" event={"ID":"19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d","Type":"ContainerStarted","Data":"d7a68b127a0fa0c20b2bf98b687f74903a62a516ee83fdb7752bc809b920d7bd"} Nov 03 00:05:25.094894 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:25.094862 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-n2pqm" Nov 03 00:05:26.273568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:26.273533 5144 patch_prober.go:29] interesting pod/image-registry-7c595ff78b-zkrdt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 00:05:26.273965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:26.273600 5144 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:05:26.976204 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:26.976168 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:26.978561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:26.978542 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1cc9f90a-15a5-41ec-9ad8-90969c74c8d4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-llwll\" (UID: \"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4\") " pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:27.016757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:27.016730 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-llwll" Nov 03 00:05:27.385301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:27.385266 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-llwll"] Nov 03 00:05:27.418557 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:27.418526 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cc9f90a_15a5_41ec_9ad8_90969c74c8d4.slice/crio-9befaac4e5b73040dc8268b5ad5ccd365d4a76d6b005b6f66b99958d0a9fa104 WatchSource:0}: Error finding container 9befaac4e5b73040dc8268b5ad5ccd365d4a76d6b005b6f66b99958d0a9fa104: Status 404 returned error can't find the container with id 9befaac4e5b73040dc8268b5ad5ccd365d4a76d6b005b6f66b99958d0a9fa104 Nov 03 00:05:27.566018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:27.565994 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-8lq8f" Nov 03 00:05:28.160240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.160206 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llwll" event={"ID":"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4","Type":"ContainerStarted","Data":"eb6a42288cd24f398e60c782414b4d50f827f829ceb0a2b244c612942b7e2bcf"} Nov 03 00:05:28.160409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.160262 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llwll" event={"ID":"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4","Type":"ContainerStarted","Data":"9befaac4e5b73040dc8268b5ad5ccd365d4a76d6b005b6f66b99958d0a9fa104"} Nov 03 00:05:28.410702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.410612 5144 patch_prober.go:29] interesting pod/image-registry-7c595ff78b-zkrdt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 00:05:28.411065 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.410701 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:05:28.518720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.518612 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-jpptx-stm4f"] Nov 03 00:05:28.527432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.527403 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.559931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.559904 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c595ff78b-zkrdt"] Nov 03 00:05:28.564440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.564417 5144 patch_prober.go:29] interesting pod/image-registry-7c595ff78b-zkrdt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 00:05:28.564572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.564472 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:05:28.587363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.587329 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.587549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.587378 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.587549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.587394 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdr88\" (UniqueName: \"kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.587682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.587574 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.688835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.688801 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.688835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.688837 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdr88\" (UniqueName: \"kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.689055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.688965 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.689055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.688995 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.689214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.689192 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.689900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.689660 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.690016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.689945 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.746521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.746484 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdr88\" (UniqueName: \"kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88\") pod \"periodic-gathering-jpptx-stm4f\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:28.775469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.775422 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jpptx-stm4f"] Nov 03 00:05:28.858084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:28.858051 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:05:29.025333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:29.025302 5144 patch_prober.go:29] interesting pod/image-registry-5cd89fb64c-8bxkm container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 00:05:29.025509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:29.025353 5144 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:05:29.166158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:29.166124 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llwll" event={"ID":"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4","Type":"ContainerStarted","Data":"cea9d968911cf4bafeff9ed6fa203a604d968557b835e2e31b0ea9d90f3e36f2"} Nov 03 00:05:29.227963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:29.227932 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-jpptx-stm4f"] Nov 03 00:05:29.230930 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:29.230904 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06647732_f09f_4779_97f2_95f895561c29.slice/crio-dc3fcfc56cf86b88fa5f625f702bf65d7ba710554490ae3810a7a7f1636fad30 WatchSource:0}: Error finding container dc3fcfc56cf86b88fa5f625f702bf65d7ba710554490ae3810a7a7f1636fad30: Status 404 returned error can't find the container with id dc3fcfc56cf86b88fa5f625f702bf65d7ba710554490ae3810a7a7f1636fad30 Nov 03 00:05:30.172584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:30.172545 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jpptx-stm4f" event={"ID":"06647732-f09f-4779-97f2-95f895561c29","Type":"ContainerStarted","Data":"7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb"} Nov 03 00:05:30.173051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:30.172608 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jpptx-stm4f" event={"ID":"06647732-f09f-4779-97f2-95f895561c29","Type":"ContainerStarted","Data":"dc3fcfc56cf86b88fa5f625f702bf65d7ba710554490ae3810a7a7f1636fad30"} Nov 03 00:05:30.285568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:30.285518 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-jpptx-stm4f" podStartSLOduration=2.285502322 podStartE2EDuration="2.285502322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:30.284564935 +0000 UTC m=+20.430259390" watchObservedRunningTime="2026-11-03 00:05:30.285502322 +0000 UTC m=+20.431196773" Nov 03 00:05:31.016926 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:31.016895 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2pqm_0964e1df-d8ed-402a-831f-b8996a1236f3/dns/0.log" Nov 03 00:05:31.178871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:31.178833 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-llwll" event={"ID":"1cc9f90a-15a5-41ec-9ad8-90969c74c8d4","Type":"ContainerStarted","Data":"d332e33aaac45d2d03a987b0ff1fc308a063a694b618d6e0db272a86cd453b8e"} Nov 03 00:05:31.180071 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:31.180050 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-n2pqm_0964e1df-d8ed-402a-831f-b8996a1236f3/kube-rbac-proxy/0.log" Nov 03 00:05:32.180013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:32.179984 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4vplk_88200eac-4b64-4019-b733-335740bfe1e5/dns-node-resolver/0.log" Nov 03 00:05:32.781318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:32.781289 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-pruner-29894400-7th8w_a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804/image-pruner/0.log" Nov 03 00:05:32.979601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:32.979566 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cd89fb64c-8bxkm_e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/registry/0.log" Nov 03 00:05:33.383407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:33.383380 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7c595ff78b-zkrdt_df250e4d-45bd-4a42-9faa-a2ed51f7589b/registry/0.log" Nov 03 00:05:33.580619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:33.580595 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-dgbsh_f0f94dde-4d2f-4969-b7ce-357282dc6529/node-ca/0.log" Nov 03 00:05:34.179667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:34.179619 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767574d6c-vss4j_31e371fb-f718-43d0-bf0f-d909838ac2d0/router/0.log" Nov 03 00:05:34.780327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:34.780296 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-xs77t_8505e4f1-35bd-4ff0-ad33-2febfff8fbf7/serve-healthcheck-canary/0.log" Nov 03 00:05:35.686042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:35.686014 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gbt9l" Nov 03 00:05:36.096497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:36.096472 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:05:38.565408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:38.565383 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:41.225107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.225078 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bdp7m"] Nov 03 00:05:41.261097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.261067 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298471 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-sys\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298507 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298531 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-root\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298590 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-wtmp\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298698 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298723 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-tls\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298748 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-textfile\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298825 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c9ab7f-0e92-400b-a88e-d4d53de42660-metrics-client-ca\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.298870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.298856 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wckd2\" (UniqueName: \"kubernetes.io/projected/29c9ab7f-0e92-400b-a88e-d4d53de42660-kube-api-access-wckd2\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.321960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.321939 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 00:05:41.322102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.321964 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 00:05:41.322241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.322219 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d8jlp\"" Nov 03 00:05:41.370562 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.370544 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 00:05:41.370663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.370618 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 00:05:41.399797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.399775 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.399887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.399806 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-tls\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.399887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.399827 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-textfile\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.399961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.399936 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c9ab7f-0e92-400b-a88e-d4d53de42660-metrics-client-ca\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.399974 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wckd2\" (UniqueName: \"kubernetes.io/projected/29c9ab7f-0e92-400b-a88e-d4d53de42660-kube-api-access-wckd2\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400030 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-sys\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400060 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400097 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-root\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400122 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-wtmp\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400257 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-textfile\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400296 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-wtmp\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400338 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-sys\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400362 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-accelerators-collector-config\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400337 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/29c9ab7f-0e92-400b-a88e-d4d53de42660-root\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.400613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.400416 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/29c9ab7f-0e92-400b-a88e-d4d53de42660-metrics-client-ca\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.402339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.402312 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-tls\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.402559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.402541 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/29c9ab7f-0e92-400b-a88e-d4d53de42660-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.456356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.456325 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wckd2\" (UniqueName: \"kubernetes.io/projected/29c9ab7f-0e92-400b-a88e-d4d53de42660-kube-api-access-wckd2\") pod \"node-exporter-bdp7m\" (UID: \"29c9ab7f-0e92-400b-a88e-d4d53de42660\") " pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.612483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:41.612454 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bdp7m" Nov 03 00:05:41.649608 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:05:41.649578 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c9ab7f_0e92_400b_a88e_d4d53de42660.slice/crio-321cd377bee2e0691227990deffe7a6c6032e914dceb5b9da8b033bacafc17d7 WatchSource:0}: Error finding container 321cd377bee2e0691227990deffe7a6c6032e914dceb5b9da8b033bacafc17d7: Status 404 returned error can't find the container with id 321cd377bee2e0691227990deffe7a6c6032e914dceb5b9da8b033bacafc17d7 Nov 03 00:05:42.152452 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:42.152420 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:05:42.224219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:42.224187 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdp7m" event={"ID":"29c9ab7f-0e92-400b-a88e-d4d53de42660","Type":"ContainerStarted","Data":"321cd377bee2e0691227990deffe7a6c6032e914dceb5b9da8b033bacafc17d7"} Nov 03 00:05:44.234166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:44.234098 5144 generic.go:356] "Generic (PLEG): container finished" podID="29c9ab7f-0e92-400b-a88e-d4d53de42660" containerID="9ccb0fe49287da11a4d1feecd2b7f40a64ef6dcef45a789f4cfebafb03e086b6" exitCode=0 Nov 03 00:05:44.234166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:44.234132 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdp7m" event={"ID":"29c9ab7f-0e92-400b-a88e-d4d53de42660","Type":"ContainerDied","Data":"9ccb0fe49287da11a4d1feecd2b7f40a64ef6dcef45a789f4cfebafb03e086b6"} Nov 03 00:05:45.240486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:45.240453 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdp7m" event={"ID":"29c9ab7f-0e92-400b-a88e-d4d53de42660","Type":"ContainerStarted","Data":"75b1fba3b9292d3fe86b19b69033f9c3184093d3e6a8410c6adaf903a8aeb164"} Nov 03 00:05:45.240866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:45.240515 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bdp7m" event={"ID":"29c9ab7f-0e92-400b-a88e-d4d53de42660","Type":"ContainerStarted","Data":"360e1ac8cc17273aa337615b5b3ce7c134e240b1775c9995e92a83c42e28d66f"} Nov 03 00:05:45.407321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:45.407271 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bdp7m" podStartSLOduration=2.184855113 podStartE2EDuration="4.40725758s" totalImagesPullingTime="2.130687817s" totalInitContainerRuntime="91.71465ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:45.406673197 +0000 UTC m=+35.552367665" watchObservedRunningTime="2026-11-03 00:05:45.40725758 +0000 UTC m=+35.552952036" Nov 03 00:05:51.050739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:51.050705 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cd89fb64c-8bxkm"] Nov 03 00:05:53.580367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.580322 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" containerID="cri-o://42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5" gracePeriod=30 Nov 03 00:05:53.810669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.810644 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:53.915268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915188 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915235 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915258 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915290 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915411 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915469 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915504 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.915556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.915534 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") pod \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\" (UID: \"df250e4d-45bd-4a42-9faa-a2ed51f7589b\") " Nov 03 00:05:53.916133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.916054 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:53.916258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.916002 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:05:53.917933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.917909 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:53.917933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.917920 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:53.918335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.918309 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:05:53.918418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.918331 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:53.918454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.918433 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "kube-api-access-82m2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:05:53.922427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:53.922402 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted" pod "df250e4d-45bd-4a42-9faa-a2ed51f7589b" (UID: "df250e4d-45bd-4a42-9faa-a2ed51f7589b"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:05:54.016655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016604 5144 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-image-registry-private-configuration\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016653 5144 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/df250e4d-45bd-4a42-9faa-a2ed51f7589b-installation-pull-secrets\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016679 5144 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-tls\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016688 5144 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-trusted-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016696 5144 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/df250e4d-45bd-4a42-9faa-a2ed51f7589b-ca-trust-extracted\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016705 5144 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/df250e4d-45bd-4a42-9faa-a2ed51f7589b-registry-certificates\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016713 5144 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-bound-sa-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.016799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.016722 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82m2t\" (UniqueName: \"kubernetes.io/projected/df250e4d-45bd-4a42-9faa-a2ed51f7589b-kube-api-access-82m2t\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:05:54.279143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.279112 5144 generic.go:356] "Generic (PLEG): container finished" podID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerID="42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5" exitCode=0 Nov 03 00:05:54.279317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.279157 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" Nov 03 00:05:54.279317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.279158 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" event={"ID":"df250e4d-45bd-4a42-9faa-a2ed51f7589b","Type":"ContainerDied","Data":"42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5"} Nov 03 00:05:54.279417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.279307 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c595ff78b-zkrdt" event={"ID":"df250e4d-45bd-4a42-9faa-a2ed51f7589b","Type":"ContainerDied","Data":"f7741e30ed6aa165676fa6a2fad10d851482dd12891fa84eb389b33e1e42c5aa"} Nov 03 00:05:54.279417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.279345 5144 scope.go:122] "RemoveContainer" containerID="42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5" Nov 03 00:05:54.289479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.289461 5144 scope.go:122] "RemoveContainer" containerID="42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5" Nov 03 00:05:54.289797 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:05:54.289772 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5\": container with ID starting with 42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5 not found: ID does not exist" containerID="42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5" Nov 03 00:05:54.289889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.289808 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5"} err="failed to get container status \"42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5\": rpc error: code = NotFound desc = could not find container \"42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5\": container with ID starting with 42a0f3fde7558a2eb14a7012986bb7362c41d85d08fe9701c6cb8ec4d5fd01d5 not found: ID does not exist" Nov 03 00:05:54.443753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.443725 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c595ff78b-zkrdt"] Nov 03 00:05:54.449057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:54.449031 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c595ff78b-zkrdt"] Nov 03 00:05:55.994699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:05:55.994671 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" path="/var/lib/kubelet/pods/df250e4d-45bd-4a42-9faa-a2ed51f7589b/volumes" Nov 03 00:06:00.408594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.408556 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rj2h8-pnnjn"] Nov 03 00:06:00.409025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.408773 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" podUID="9425f6e8-123b-4acb-a3d1-5ccd19761401" containerName="insights-gathering" containerID="cri-o://ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167" gracePeriod=30 Nov 03 00:06:00.658833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.658767 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:06:00.773788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.773756 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") pod \"9425f6e8-123b-4acb-a3d1-5ccd19761401\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " Nov 03 00:06:00.773961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.773836 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") pod \"9425f6e8-123b-4acb-a3d1-5ccd19761401\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " Nov 03 00:06:00.773961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.773864 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") pod \"9425f6e8-123b-4acb-a3d1-5ccd19761401\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " Nov 03 00:06:00.773961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.773900 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") pod \"9425f6e8-123b-4acb-a3d1-5ccd19761401\" (UID: \"9425f6e8-123b-4acb-a3d1-5ccd19761401\") " Nov 03 00:06:00.774200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.774173 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path" pod "9425f6e8-123b-4acb-a3d1-5ccd19761401" (UID: "9425f6e8-123b-4acb-a3d1-5ccd19761401"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:00.774261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.774196 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle" pod "9425f6e8-123b-4acb-a3d1-5ccd19761401" (UID: "9425f6e8-123b-4acb-a3d1-5ccd19761401"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:00.774261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.774206 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle" pod "9425f6e8-123b-4acb-a3d1-5ccd19761401" (UID: "9425f6e8-123b-4acb-a3d1-5ccd19761401"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:00.776880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.776851 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt" pod "9425f6e8-123b-4acb-a3d1-5ccd19761401" (UID: "9425f6e8-123b-4acb-a3d1-5ccd19761401"). InnerVolumeSpecName "kube-api-access-dqsbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:00.874418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.874372 5144 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-service-ca-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:00.874418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.874412 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqsbt\" (UniqueName: \"kubernetes.io/projected/9425f6e8-123b-4acb-a3d1-5ccd19761401-kube-api-access-dqsbt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:00.874418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.874441 5144 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/9425f6e8-123b-4acb-a3d1-5ccd19761401-archives-path\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:00.874716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:00.874452 5144 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9425f6e8-123b-4acb-a3d1-5ccd19761401-trusted-ca-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:01.309784 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.309699 5144 generic.go:356] "Generic (PLEG): container finished" podID="9425f6e8-123b-4acb-a3d1-5ccd19761401" containerID="ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167" exitCode=2 Nov 03 00:06:01.309784 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.309747 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" Nov 03 00:06:01.309970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.309774 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" event={"ID":"9425f6e8-123b-4acb-a3d1-5ccd19761401","Type":"ContainerDied","Data":"ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167"} Nov 03 00:06:01.309970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.309829 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-rj2h8-pnnjn" event={"ID":"9425f6e8-123b-4acb-a3d1-5ccd19761401","Type":"ContainerDied","Data":"26d3a6eade45c27395a68ee238de5a3978df1b7685924d99884b91146c81e22f"} Nov 03 00:06:01.309970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.309852 5144 scope.go:122] "RemoveContainer" containerID="ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167" Nov 03 00:06:01.322753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.322729 5144 scope.go:122] "RemoveContainer" containerID="ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167" Nov 03 00:06:01.323112 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:06:01.323092 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167\": container with ID starting with ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167 not found: ID does not exist" containerID="ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167" Nov 03 00:06:01.323165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.323122 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167"} err="failed to get container status \"ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167\": rpc error: code = NotFound desc = could not find container \"ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167\": container with ID starting with ae39bff7fbedbfea5c155eb68f7e8958bf7b1b501722bf0832b8da473fe2b167 not found: ID does not exist" Nov 03 00:06:01.471602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.471571 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-rj2h8-pnnjn"] Nov 03 00:06:01.520923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.520884 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-rj2h8-pnnjn"] Nov 03 00:06:01.994013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:01.993978 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9425f6e8-123b-4acb-a3d1-5ccd19761401" path="/var/lib/kubelet/pods/9425f6e8-123b-4acb-a3d1-5ccd19761401/volumes" Nov 03 00:06:12.155360 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:06:12.155317 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:06:16.074001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.073941 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerName="registry" containerID="cri-o://12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b" gracePeriod=30 Nov 03 00:06:16.092725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.092692 5144 patch_prober.go:29] interesting pod/image-registry-5cd89fb64c-8bxkm container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.132.0.16:5000/healthz\": dial tcp 10.132.0.16:5000: connect: connection refused" start-of-body= Nov 03 00:06:16.092848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.092756 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerName="registry" probeResult="failure" output="Get \"https://10.132.0.16:5000/healthz\": dial tcp 10.132.0.16:5000: connect: connection refused" Nov 03 00:06:16.323935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.323900 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:06:16.369113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.369023 5144 generic.go:356] "Generic (PLEG): container finished" podID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerID="12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b" exitCode=0 Nov 03 00:06:16.369113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.369073 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" Nov 03 00:06:16.369282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.369077 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" event={"ID":"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a","Type":"ContainerDied","Data":"12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b"} Nov 03 00:06:16.369282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.369186 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cd89fb64c-8bxkm" event={"ID":"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a","Type":"ContainerDied","Data":"d614b04b922a9a43db2eb85d6c52e178c397d65793703f74844031e199b9c90d"} Nov 03 00:06:16.369282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.369200 5144 scope.go:122] "RemoveContainer" containerID="12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b" Nov 03 00:06:16.377381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.377362 5144 scope.go:122] "RemoveContainer" containerID="12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b" Nov 03 00:06:16.377745 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:06:16.377709 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b\": container with ID starting with 12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b not found: ID does not exist" containerID="12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b" Nov 03 00:06:16.377804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.377755 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b"} err="failed to get container status \"12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b\": rpc error: code = NotFound desc = could not find container \"12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b\": container with ID starting with 12f654abeab7d4f59be9c2460a58612fb0f97d1b6d97227ad176df2722484b7b not found: ID does not exist" Nov 03 00:06:16.501659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501592 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.501846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501674 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.501846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501707 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.501846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501735 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.501846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501783 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.501846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501812 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.501846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.501842 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.502140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.502094 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") pod \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\" (UID: \"e7ac0dbb-01b6-4660-abca-8b8c2a7d741a\") " Nov 03 00:06:16.502473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.502411 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:16.502957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.502908 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:16.504749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.504709 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:16.505101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.505067 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:16.505404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.505123 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "kube-api-access-hgpgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:16.505404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.505126 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:16.505404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.505243 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:06:16.511707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.511663 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted" pod "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" (UID: "e7ac0dbb-01b6-4660-abca-8b8c2a7d741a"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:16.603362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603323 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgpgv\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-kube-api-access-hgpgv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603356 5144 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-tls\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603366 5144 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-installation-pull-secrets\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603376 5144 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-image-registry-private-configuration\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603386 5144 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-ca-trust-extracted\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603395 5144 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-trusted-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603404 5144 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-bound-sa-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.603615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.603414 5144 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a-registry-certificates\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:16.786597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.786561 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cd89fb64c-8bxkm"] Nov 03 00:06:16.790426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:16.790394 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cd89fb64c-8bxkm"] Nov 03 00:06:17.993779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:17.993745 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" path="/var/lib/kubelet/pods/e7ac0dbb-01b6-4660-abca-8b8c2a7d741a/volumes" Nov 03 00:06:23.399438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:23.399400 5144 generic.go:356] "Generic (PLEG): container finished" podID="f58b544f-c636-4c93-9e1c-9a817590d307" containerID="4a10bd910d35780899a2e0ffbeaedc4fc48df054c2d95f33913e36139c93d4e4" exitCode=0 Nov 03 00:06:23.399938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:23.399469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mdw87" event={"ID":"f58b544f-c636-4c93-9e1c-9a817590d307","Type":"ContainerDied","Data":"4a10bd910d35780899a2e0ffbeaedc4fc48df054c2d95f33913e36139c93d4e4"} Nov 03 00:06:23.399938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:23.399527 5144 scope.go:122] "RemoveContainer" containerID="11cbea1dfb216d6eef0305df8d92fb9e5ab9cf85c29a209fc0a9be03aaba9fa2" Nov 03 00:06:23.399938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:23.399889 5144 scope.go:122] "RemoveContainer" containerID="4a10bd910d35780899a2e0ffbeaedc4fc48df054c2d95f33913e36139c93d4e4" Nov 03 00:06:23.400161 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:06:23.400142 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"insights-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=insights-operator pod=insights-operator-7c4986d449-mdw87_openshift-insights(f58b544f-c636-4c93-9e1c-9a817590d307)\"" pod="openshift-insights/insights-operator-7c4986d449-mdw87" podUID="f58b544f-c636-4c93-9e1c-9a817590d307" Nov 03 00:06:31.433052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:31.433018 5144 generic.go:356] "Generic (PLEG): container finished" podID="a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" containerID="1f47ceb8ed88708d7d92561f3404ddca5575ddf2bd203bea56bbd9f1422243bd" exitCode=0 Nov 03 00:06:31.433463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:31.433092 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-7th8w" event={"ID":"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804","Type":"ContainerDied","Data":"1f47ceb8ed88708d7d92561f3404ddca5575ddf2bd203bea56bbd9f1422243bd"} Nov 03 00:06:32.574140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.574113 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:06:32.630830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.630796 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") pod \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " Nov 03 00:06:32.631029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.630870 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") pod \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\" (UID: \"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804\") " Nov 03 00:06:32.631208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.631184 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca" pod "a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" (UID: "a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804"). InnerVolumeSpecName "serviceca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:32.633200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.633174 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7" pod "a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" (UID: "a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804"). InnerVolumeSpecName "kube-api-access-krrk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:32.732024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.731924 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krrk7\" (UniqueName: \"kubernetes.io/projected/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-kube-api-access-krrk7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:32.732024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:32.731961 5144 reconciler_common.go:299] "Volume detached for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804-serviceca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:33.441754 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:33.441719 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-pruner-29894400-7th8w" event={"ID":"a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804","Type":"ContainerDied","Data":"ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262"} Nov 03 00:06:33.441906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:33.441766 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea08a296039010fc947b3949a02037a7375f777cf3386581945cf0217a0ae262" Nov 03 00:06:33.441906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:33.441736 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-pruner-29894400-7th8w" Nov 03 00:06:35.989412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:35.989375 5144 scope.go:122] "RemoveContainer" containerID="4a10bd910d35780899a2e0ffbeaedc4fc48df054c2d95f33913e36139c93d4e4" Nov 03 00:06:36.458953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:36.458916 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-mdw87" event={"ID":"f58b544f-c636-4c93-9e1c-9a817590d307","Type":"ContainerStarted","Data":"0a0e7181cc330056b12b88e5627894fca45f02d7f1360425cba16e0c37466496"} Nov 03 00:06:37.001651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.001586 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jpptx-stm4f"] Nov 03 00:06:37.002063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.001806 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-jpptx-stm4f" podUID="06647732-f09f-4779-97f2-95f895561c29" containerName="insights-gathering" containerID="cri-o://7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb" gracePeriod=30 Nov 03 00:06:37.257433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.257398 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:06:37.372543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.372511 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path\") pod \"06647732-f09f-4779-97f2-95f895561c29\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " Nov 03 00:06:37.372762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.372561 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle\") pod \"06647732-f09f-4779-97f2-95f895561c29\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " Nov 03 00:06:37.372762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.372582 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88\" (UniqueName: \"kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88\") pod \"06647732-f09f-4779-97f2-95f895561c29\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " Nov 03 00:06:37.372762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.372674 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle\") pod \"06647732-f09f-4779-97f2-95f895561c29\" (UID: \"06647732-f09f-4779-97f2-95f895561c29\") " Nov 03 00:06:37.372958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.372929 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path" pod "06647732-f09f-4779-97f2-95f895561c29" (UID: "06647732-f09f-4779-97f2-95f895561c29"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:06:37.373015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.372950 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle" pod "06647732-f09f-4779-97f2-95f895561c29" (UID: "06647732-f09f-4779-97f2-95f895561c29"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:37.373215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.373189 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle" pod "06647732-f09f-4779-97f2-95f895561c29" (UID: "06647732-f09f-4779-97f2-95f895561c29"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:06:37.375070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.375039 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88" pod "06647732-f09f-4779-97f2-95f895561c29" (UID: "06647732-f09f-4779-97f2-95f895561c29"). InnerVolumeSpecName "kube-api-access-rdr88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:06:37.464225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.464186 5144 generic.go:356] "Generic (PLEG): container finished" podID="06647732-f09f-4779-97f2-95f895561c29" containerID="7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb" exitCode=2 Nov 03 00:06:37.464404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.464231 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-jpptx-stm4f" Nov 03 00:06:37.464404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.464276 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jpptx-stm4f" event={"ID":"06647732-f09f-4779-97f2-95f895561c29","Type":"ContainerDied","Data":"7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb"} Nov 03 00:06:37.464404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.464326 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-jpptx-stm4f" event={"ID":"06647732-f09f-4779-97f2-95f895561c29","Type":"ContainerDied","Data":"dc3fcfc56cf86b88fa5f625f702bf65d7ba710554490ae3810a7a7f1636fad30"} Nov 03 00:06:37.464404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.464348 5144 scope.go:122] "RemoveContainer" containerID="7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb" Nov 03 00:06:37.473154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.473135 5144 scope.go:122] "RemoveContainer" containerID="7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb" Nov 03 00:06:37.473502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.473483 5144 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/06647732-f09f-4779-97f2-95f895561c29-archives-path\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:37.473559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.473511 5144 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-service-ca-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:37.473559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.473523 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdr88\" (UniqueName: \"kubernetes.io/projected/06647732-f09f-4779-97f2-95f895561c29-kube-api-access-rdr88\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:37.473559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.473532 5144 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/06647732-f09f-4779-97f2-95f895561c29-trusted-ca-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:06:37.473559 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:06:37.473490 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb\": container with ID starting with 7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb not found: ID does not exist" containerID="7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb" Nov 03 00:06:37.473722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.473558 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb"} err="failed to get container status \"7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb\": rpc error: code = NotFound desc = could not find container \"7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb\": container with ID starting with 7ed69872455214756c2d413b64aeace91db00624c8561d50f34919dffcd8f4cb not found: ID does not exist" Nov 03 00:06:37.627136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.627042 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-jpptx-stm4f"] Nov 03 00:06:37.632653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.632616 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-jpptx-stm4f"] Nov 03 00:06:37.993186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:37.993097 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06647732-f09f-4779-97f2-95f895561c29" path="/var/lib/kubelet/pods/06647732-f09f-4779-97f2-95f895561c29/volumes" Nov 03 00:06:39.474206 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:39.474172 5144 generic.go:356] "Generic (PLEG): container finished" podID="33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59" containerID="a97ecefc68afeeaeb689cdf2c4fd077d1da3d290e9460340d0ab6cca7cd1426d" exitCode=0 Nov 03 00:06:39.474616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:39.474243 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" event={"ID":"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59","Type":"ContainerDied","Data":"a97ecefc68afeeaeb689cdf2c4fd077d1da3d290e9460340d0ab6cca7cd1426d"} Nov 03 00:06:39.474616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:39.474603 5144 scope.go:122] "RemoveContainer" containerID="a97ecefc68afeeaeb689cdf2c4fd077d1da3d290e9460340d0ab6cca7cd1426d" Nov 03 00:06:40.479569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:40.479481 5144 generic.go:356] "Generic (PLEG): container finished" podID="745ded03-fed2-450a-bb70-8a78b9d3200d" containerID="e59698448079a90b47e802ecc947a2d107959ef018fe1401fd7398a57ce44a4e" exitCode=0 Nov 03 00:06:40.480028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:40.479559 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" event={"ID":"745ded03-fed2-450a-bb70-8a78b9d3200d","Type":"ContainerDied","Data":"e59698448079a90b47e802ecc947a2d107959ef018fe1401fd7398a57ce44a4e"} Nov 03 00:06:40.480028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:40.479935 5144 scope.go:122] "RemoveContainer" containerID="e59698448079a90b47e802ecc947a2d107959ef018fe1401fd7398a57ce44a4e" Nov 03 00:06:40.481617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:40.481595 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-fgn2w" event={"ID":"33e4c7ae-33c0-4b0d-a1d9-bda9f56a5d59","Type":"ContainerStarted","Data":"74af8ed42c71164eb41844549081d11450276c8611f53c3a42abf1f8f5e1acb3"} Nov 03 00:06:41.487061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:06:41.487017 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-xxhj6" event={"ID":"745ded03-fed2-450a-bb70-8a78b9d3200d","Type":"ContainerStarted","Data":"6a9bea505adbd8e1460439d0db58a0872a2d854958b1369e8a425f9fc056c63f"} Nov 03 00:06:42.154347 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:06:42.154312 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:07:12.154465 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:07:12.154426 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:07:42.154187 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:07:42.154151 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:08:07.951722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.951680 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw"] Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952112 5144 cpu_manager.go:414] "containerMap: removing container" podUID="06647732-f09f-4779-97f2-95f895561c29" containerName="insights-gathering" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952130 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06647732-f09f-4779-97f2-95f895561c29" containerName="insights-gathering" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952149 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" containerName="image-pruner" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952156 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" containerName="image-pruner" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952174 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerName="registry" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952181 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerName="registry" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952189 5144 cpu_manager.go:414] "containerMap: removing container" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952195 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952226 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9425f6e8-123b-4acb-a3d1-5ccd19761401" containerName="insights-gathering" Nov 03 00:08:07.952278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952234 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9425f6e8-123b-4acb-a3d1-5ccd19761401" containerName="insights-gathering" Nov 03 00:08:07.952831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952322 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7ac0dbb-01b6-4660-abca-8b8c2a7d741a" containerName="registry" Nov 03 00:08:07.952831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952333 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="df250e4d-45bd-4a42-9faa-a2ed51f7589b" containerName="registry" Nov 03 00:08:07.952831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952347 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9425f6e8-123b-4acb-a3d1-5ccd19761401" containerName="insights-gathering" Nov 03 00:08:07.952831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952357 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="06647732-f09f-4779-97f2-95f895561c29" containerName="insights-gathering" Nov 03 00:08:07.952831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.952365 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a84c8cd4-9ec7-4ae7-bc01-64a0b37f9804" containerName="image-pruner" Nov 03 00:08:07.955709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:07.955689 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.011002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.010969 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw"] Nov 03 00:08:08.059299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.059267 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Nov 03 00:08:08.059299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.059267 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Nov 03 00:08:08.059880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.059860 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.059930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.059896 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t58rg\" (UniqueName: \"kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.059999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.059983 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.062414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.062399 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-n4zlt\"" Nov 03 00:08:08.063505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.063486 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl"] Nov 03 00:08:08.066931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.066915 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.120131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.120097 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl"] Nov 03 00:08:08.122779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.122750 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-v7dc2\"" Nov 03 00:08:08.158387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.158354 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2"] Nov 03 00:08:08.160490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160467 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.160590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160500 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.160590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160533 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.160713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160621 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t58rg\" (UniqueName: \"kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.160771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160705 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k9jm\" (UniqueName: \"kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.160823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160776 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.160950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.160932 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.161102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.161083 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.161847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.161832 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.216812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.216734 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t58rg\" (UniqueName: \"kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.220249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.220217 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2"] Nov 03 00:08:08.224587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.224563 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-9d2fz\"" Nov 03 00:08:08.261251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261209 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k9jm\" (UniqueName: \"kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.261420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261273 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.261420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261310 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcwp4\" (UniqueName: \"kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.261420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261346 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.261583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261418 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.261583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261534 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.261812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261793 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.261897 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.261862 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.271170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.271153 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:08.318069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.318012 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k9jm\" (UniqueName: \"kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.362459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.362425 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.362615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.362470 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcwp4\" (UniqueName: \"kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.362615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.362506 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.362920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.362896 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.362960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.362901 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.382359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.382326 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:08.418897 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.418868 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcwp4\" (UniqueName: \"kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.477221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.477149 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:08.639226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.639203 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw"] Nov 03 00:08:08.641787 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:08.641759 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd29826_fbc5_46e4_8333_045d9f7cced3.slice/crio-6eb4b7cc133fe07c5b76f8ad26f0bb93b992c08879689e2b955e4c07baa254bd WatchSource:0}: Error finding container 6eb4b7cc133fe07c5b76f8ad26f0bb93b992c08879689e2b955e4c07baa254bd: Status 404 returned error can't find the container with id 6eb4b7cc133fe07c5b76f8ad26f0bb93b992c08879689e2b955e4c07baa254bd Nov 03 00:08:08.750302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.750218 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl"] Nov 03 00:08:08.753156 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:08.753118 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8d8690_756f_4734_98fa_9e4b1e610053.slice/crio-089ea798e18bd36958a05b999d54204d08f10e9081968eb7748d34c5666ea3e2 WatchSource:0}: Error finding container 089ea798e18bd36958a05b999d54204d08f10e9081968eb7748d34c5666ea3e2: Status 404 returned error can't find the container with id 089ea798e18bd36958a05b999d54204d08f10e9081968eb7748d34c5666ea3e2 Nov 03 00:08:08.833945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.833901 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" event={"ID":"6fd29826-fbc5-46e4-8333-045d9f7cced3","Type":"ContainerStarted","Data":"6eb4b7cc133fe07c5b76f8ad26f0bb93b992c08879689e2b955e4c07baa254bd"} Nov 03 00:08:08.835052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.835029 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" event={"ID":"5c8d8690-756f-4734-98fa-9e4b1e610053","Type":"ContainerStarted","Data":"089ea798e18bd36958a05b999d54204d08f10e9081968eb7748d34c5666ea3e2"} Nov 03 00:08:08.843716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:08.843695 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2"] Nov 03 00:08:08.846026 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:08.846006 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07a95090_0b1a_4941_a1da_6cc79a48125f.slice/crio-7e757d58897c86673e463e6aadbe3c9adba13ffd5687aa3efff13a145d4ebecc WatchSource:0}: Error finding container 7e757d58897c86673e463e6aadbe3c9adba13ffd5687aa3efff13a145d4ebecc: Status 404 returned error can't find the container with id 7e757d58897c86673e463e6aadbe3c9adba13ffd5687aa3efff13a145d4ebecc Nov 03 00:08:09.479904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.479870 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567"] Nov 03 00:08:09.484876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.484855 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.540259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.540229 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567"] Nov 03 00:08:09.572460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.572415 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.572646 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.572481 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrk2t\" (UniqueName: \"kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.572646 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.572528 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.588241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.588207 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-9wzcp\"" Nov 03 00:08:09.673753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.673498 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.673753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.673565 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk2t\" (UniqueName: \"kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.673753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.673617 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.673995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.673938 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.674051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.673994 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.731271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.731197 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk2t\" (UniqueName: \"kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.801203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.801167 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:09.844596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:09.844560 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" event={"ID":"07a95090-0b1a-4941-a1da-6cc79a48125f","Type":"ContainerStarted","Data":"7e757d58897c86673e463e6aadbe3c9adba13ffd5687aa3efff13a145d4ebecc"} Nov 03 00:08:10.192530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.192452 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567"] Nov 03 00:08:10.195566 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:10.195541 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86fa780e_92a2_4f70_ab20_41291309f56b.slice/crio-6f469191f2fb082ff5bc9117e56ca70c22bbac772df4d765ad271e38024673d5 WatchSource:0}: Error finding container 6f469191f2fb082ff5bc9117e56ca70c22bbac772df4d765ad271e38024673d5: Status 404 returned error can't find the container with id 6f469191f2fb082ff5bc9117e56ca70c22bbac772df4d765ad271e38024673d5 Nov 03 00:08:10.279767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.279732 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4"] Nov 03 00:08:10.285001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.284971 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.339949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.339915 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-pqhzv\"" Nov 03 00:08:10.341939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.341913 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4"] Nov 03 00:08:10.379660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.379448 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.379660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.379520 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.379660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.379586 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwx8m\" (UniqueName: \"kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.481044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.480952 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.481044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.481021 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.481595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.481086 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwx8m\" (UniqueName: \"kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.481851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.481827 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.481976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.481935 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.538555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.538518 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwx8m\" (UniqueName: \"kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.604440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.604395 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:10.851125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.851024 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" event={"ID":"86fa780e-92a2-4f70-ab20-41291309f56b","Type":"ContainerStarted","Data":"6f469191f2fb082ff5bc9117e56ca70c22bbac772df4d765ad271e38024673d5"} Nov 03 00:08:10.994893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:10.994854 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4"] Nov 03 00:08:10.999393 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:10.999360 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f8c8aa1_8cd3_4aa4_b306_d27ccdab5443.slice/crio-143ca4fea6781d5448f2ef2bef279e1418d9fee71f4ec6ba55759bbd1d664960 WatchSource:0}: Error finding container 143ca4fea6781d5448f2ef2bef279e1418d9fee71f4ec6ba55759bbd1d664960: Status 404 returned error can't find the container with id 143ca4fea6781d5448f2ef2bef279e1418d9fee71f4ec6ba55759bbd1d664960 Nov 03 00:08:11.860677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:11.860609 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" event={"ID":"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443","Type":"ContainerStarted","Data":"143ca4fea6781d5448f2ef2bef279e1418d9fee71f4ec6ba55759bbd1d664960"} Nov 03 00:08:12.200383 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:08:12.200289 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:08:13.160952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.160917 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478"] Nov 03 00:08:13.165021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.164979 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.208372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.208082 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.208372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.208138 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.208372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.208187 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-br4vf\" (UniqueName: \"kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.220300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.220240 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9khbq\"" Nov 03 00:08:13.222485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.222402 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478"] Nov 03 00:08:13.309766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.308802 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.309766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.308846 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.309766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.308922 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-br4vf\" (UniqueName: \"kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.309766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.309714 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.310183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.310160 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.373724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.373647 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-br4vf\" (UniqueName: \"kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.489610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.489266 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:13.918060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:13.917668 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478"] Nov 03 00:08:13.923835 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:13.923791 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e18ec5f_d608_413d_b3f7_610a1de01da1.slice/crio-84b8da9b953878a82e498455a72d03ffc794974764beb4aafa9a48c1ce89471d WatchSource:0}: Error finding container 84b8da9b953878a82e498455a72d03ffc794974764beb4aafa9a48c1ce89471d: Status 404 returned error can't find the container with id 84b8da9b953878a82e498455a72d03ffc794974764beb4aafa9a48c1ce89471d Nov 03 00:08:14.881447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:14.881351 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" event={"ID":"5e18ec5f-d608-413d-b3f7-610a1de01da1","Type":"ContainerStarted","Data":"84b8da9b953878a82e498455a72d03ffc794974764beb4aafa9a48c1ce89471d"} Nov 03 00:08:16.717360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.715650 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2"] Nov 03 00:08:16.720679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.720620 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.776669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.775326 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 00:08:16.822436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.822388 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 00:08:16.824800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.824763 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2"] Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843650 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843726 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843762 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843801 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843841 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843867 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843926 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrm48\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843967 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.844156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.843985 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.944990 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945046 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945086 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945121 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945154 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945187 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945224 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945249 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.945307 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrm48\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.946085 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.946094 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.946802 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.947075 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.947707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.947317 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.956291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.956232 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:16.960337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:16.960299 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:17.007666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.007385 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrm48\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:17.011960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.011883 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-qffb2\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:17.016473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.016410 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4"] Nov 03 00:08:17.022131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.022098 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.039820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.039779 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:17.080760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.080230 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4"] Nov 03 00:08:17.080760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.080423 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-6zj6z\"" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147060 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbl6p\" (UniqueName: \"kubernetes.io/projected/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-kube-api-access-kbl6p\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147114 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147159 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147195 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147215 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147233 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147282 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147302 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.147462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.147321 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248307 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbl6p\" (UniqueName: \"kubernetes.io/projected/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-kube-api-access-kbl6p\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248394 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248446 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248486 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248516 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248548 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248622 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248675 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.248887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.248710 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.249171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.249141 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.250412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.250385 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.250662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.250625 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.251569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.251279 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.251569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.251529 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.253197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.253160 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.253521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.253501 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.308913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.308792 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.311909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.311875 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbl6p\" (UniqueName: \"kubernetes.io/projected/c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa-kube-api-access-kbl6p\") pod \"openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4\" (UID: \"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.347132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.346732 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:17.525584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.525542 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2"] Nov 03 00:08:17.543344 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:17.543264 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod281353eb_04c5_4896_bf1e_b6948829ac54.slice/crio-e4d30ea589977a9a5c4fc7e707bb9810cabab909dbce1c629b85203678a17a9c WatchSource:0}: Error finding container e4d30ea589977a9a5c4fc7e707bb9810cabab909dbce1c629b85203678a17a9c: Status 404 returned error can't find the container with id e4d30ea589977a9a5c4fc7e707bb9810cabab909dbce1c629b85203678a17a9c Nov 03 00:08:17.771487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.771312 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4"] Nov 03 00:08:17.775418 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:17.775370 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7ce7db6_4c1e_4dce_a4cf_c735143ee8fa.slice/crio-93abacd2330c7f08fa0a9d760a1c0da3db402b93b3775316f9d957cf966baa71 WatchSource:0}: Error finding container 93abacd2330c7f08fa0a9d760a1c0da3db402b93b3775316f9d957cf966baa71: Status 404 returned error can't find the container with id 93abacd2330c7f08fa0a9d760a1c0da3db402b93b3775316f9d957cf966baa71 Nov 03 00:08:17.909835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.909788 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" event={"ID":"281353eb-04c5-4896-bf1e-b6948829ac54","Type":"ContainerStarted","Data":"e4d30ea589977a9a5c4fc7e707bb9810cabab909dbce1c629b85203678a17a9c"} Nov 03 00:08:17.912310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:17.912272 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" event={"ID":"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa","Type":"ContainerStarted","Data":"93abacd2330c7f08fa0a9d760a1c0da3db402b93b3775316f9d957cf966baa71"} Nov 03 00:08:19.929248 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:19.928979 5144 generic.go:356] "Generic (PLEG): container finished" podID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerID="c56de5424091a5dadb73921d82b95b979d1ed3e4589a91a41491af42f862eaab" exitCode=0 Nov 03 00:08:19.929248 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:19.929131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" event={"ID":"6fd29826-fbc5-46e4-8333-045d9f7cced3","Type":"ContainerDied","Data":"c56de5424091a5dadb73921d82b95b979d1ed3e4589a91a41491af42f862eaab"} Nov 03 00:08:19.932755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:19.932713 5144 generic.go:356] "Generic (PLEG): container finished" podID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerID="c75aee2ac9d260292700f123dac5dcb903f58bab7607dc95fcf6565d44163ce6" exitCode=0 Nov 03 00:08:19.932868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:19.932822 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" event={"ID":"5c8d8690-756f-4734-98fa-9e4b1e610053","Type":"ContainerDied","Data":"c75aee2ac9d260292700f123dac5dcb903f58bab7607dc95fcf6565d44163ce6"} Nov 03 00:08:21.946319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:21.946281 5144 generic.go:356] "Generic (PLEG): container finished" podID="86fa780e-92a2-4f70-ab20-41291309f56b" containerID="ea37f1f3025f2dd3933cf3c4896bf5e5f0ddbb492b9d546dcbdea63096b13860" exitCode=0 Nov 03 00:08:21.946786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:21.946547 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" event={"ID":"86fa780e-92a2-4f70-ab20-41291309f56b","Type":"ContainerDied","Data":"ea37f1f3025f2dd3933cf3c4896bf5e5f0ddbb492b9d546dcbdea63096b13860"} Nov 03 00:08:21.948964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:21.948941 5144 generic.go:356] "Generic (PLEG): container finished" podID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerID="45b986a4db3fe2bb87709078d1ac57a7689ed58f4f8b1487427d71c5ef04cd3b" exitCode=0 Nov 03 00:08:21.949100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:21.949027 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" event={"ID":"07a95090-0b1a-4941-a1da-6cc79a48125f","Type":"ContainerDied","Data":"45b986a4db3fe2bb87709078d1ac57a7689ed58f4f8b1487427d71c5ef04cd3b"} Nov 03 00:08:21.951229 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:21.951205 5144 generic.go:356] "Generic (PLEG): container finished" podID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerID="95f32faf777adf15814319391a84acc70c3163fb3b48b6a7637067ce6f4c6319" exitCode=0 Nov 03 00:08:21.951344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:21.951274 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" event={"ID":"5e18ec5f-d608-413d-b3f7-610a1de01da1","Type":"ContainerDied","Data":"95f32faf777adf15814319391a84acc70c3163fb3b48b6a7637067ce6f4c6319"} Nov 03 00:08:22.963224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:22.962176 5144 generic.go:356] "Generic (PLEG): container finished" podID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerID="0ed95835221ea346b53e8ebad8554822a33154feba97146edc9a7b5a59933b6c" exitCode=0 Nov 03 00:08:22.963224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:22.962311 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" event={"ID":"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443","Type":"ContainerDied","Data":"0ed95835221ea346b53e8ebad8554822a33154feba97146edc9a7b5a59933b6c"} Nov 03 00:08:22.966560 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:22.966523 5144 generic.go:356] "Generic (PLEG): container finished" podID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerID="8903c222d9d8584737eaae4a3d015845ff1a2f0561406ba3ac2dba27d220fea6" exitCode=0 Nov 03 00:08:22.966698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:22.966657 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" event={"ID":"5e18ec5f-d608-413d-b3f7-610a1de01da1","Type":"ContainerDied","Data":"8903c222d9d8584737eaae4a3d015845ff1a2f0561406ba3ac2dba27d220fea6"} Nov 03 00:08:24.980398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:24.980183 5144 generic.go:356] "Generic (PLEG): container finished" podID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerID="9a3022cc708a537634c9cb9e3d5caa2c5a89f66184e33da68562f1670b1cf713" exitCode=0 Nov 03 00:08:24.980398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:24.980241 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" event={"ID":"5c8d8690-756f-4734-98fa-9e4b1e610053","Type":"ContainerDied","Data":"9a3022cc708a537634c9cb9e3d5caa2c5a89f66184e33da68562f1670b1cf713"} Nov 03 00:08:24.984280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:24.984258 5144 generic.go:356] "Generic (PLEG): container finished" podID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerID="1627238262fe0cebf71a081d399c66420ee6d5c74a0fb5abdf838e148ae07b21" exitCode=0 Nov 03 00:08:24.984395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:24.984321 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" event={"ID":"6fd29826-fbc5-46e4-8333-045d9f7cced3","Type":"ContainerDied","Data":"1627238262fe0cebf71a081d399c66420ee6d5c74a0fb5abdf838e148ae07b21"} Nov 03 00:08:25.469058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.469020 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:08:25.469189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.469113 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:08:25.469189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.469156 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:08:25.729074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.729038 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:08:25.729203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.729146 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:08:25.729203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.729180 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:08:25.997603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:25.997528 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" event={"ID":"281353eb-04c5-4896-bf1e-b6948829ac54","Type":"ContainerStarted","Data":"2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c"} Nov 03 00:08:26.002055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.002021 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" event={"ID":"c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa","Type":"ContainerStarted","Data":"e268044a163708f9df59acb688e97cbc823c0e69e7fbb085da97c1ea5a5b0db3"} Nov 03 00:08:26.040668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.040621 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:26.042170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.042117 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 00:08:26.042303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.042173 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 00:08:26.112221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.112158 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podStartSLOduration=1.931344535 podStartE2EDuration="10.112139501s" totalImagesPullingTime="8.180794966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:26.111546699 +0000 UTC m=+196.257241178" watchObservedRunningTime="2026-11-03 00:08:26.112139501 +0000 UTC m=+196.257833956" Nov 03 00:08:26.227979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.227929 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" podStartSLOduration=2.538301035 podStartE2EDuration="10.227913533s" totalImagesPullingTime="7.689612498s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:26.227218769 +0000 UTC m=+196.372913251" watchObservedRunningTime="2026-11-03 00:08:26.227913533 +0000 UTC m=+196.373607986" Nov 03 00:08:26.349033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.348972 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:26.356739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:26.356688 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:27.009867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:27.009837 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:27.010792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:27.010767 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4" Nov 03 00:08:27.045691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:27.045611 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:28.013785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:28.013749 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:28.015333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:28.015109 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:08:29.020795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:29.020757 5144 generic.go:356] "Generic (PLEG): container finished" podID="86fa780e-92a2-4f70-ab20-41291309f56b" containerID="6ce14d2233654f3670e91a0ebf2a05f5556d81c66635eb822bd80de10ee22e77" exitCode=0 Nov 03 00:08:29.021279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:29.020978 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" event={"ID":"86fa780e-92a2-4f70-ab20-41291309f56b","Type":"ContainerDied","Data":"6ce14d2233654f3670e91a0ebf2a05f5556d81c66635eb822bd80de10ee22e77"} Nov 03 00:08:30.028688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:30.028650 5144 generic.go:356] "Generic (PLEG): container finished" podID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerID="aae1b222491b582904e6acdb21b6e903137b37f3e81f23cc4c1a650f8766620e" exitCode=0 Nov 03 00:08:30.029159 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:30.028736 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" event={"ID":"07a95090-0b1a-4941-a1da-6cc79a48125f","Type":"ContainerDied","Data":"aae1b222491b582904e6acdb21b6e903137b37f3e81f23cc4c1a650f8766620e"} Nov 03 00:08:33.058303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:33.058248 5144 generic.go:356] "Generic (PLEG): container finished" podID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerID="7737117f13c2ea5a26f47d9710a82cdb08ad2f7f5a3b43ea4ba77a1a4f122832" exitCode=0 Nov 03 00:08:33.059073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:33.058347 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" event={"ID":"5e18ec5f-d608-413d-b3f7-610a1de01da1","Type":"ContainerDied","Data":"7737117f13c2ea5a26f47d9710a82cdb08ad2f7f5a3b43ea4ba77a1a4f122832"} Nov 03 00:08:34.065965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.065928 5144 generic.go:356] "Generic (PLEG): container finished" podID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerID="4ccd3099c633cbf55668cf9097130eea9ad521150e62b07873250a9bf3e0520a" exitCode=0 Nov 03 00:08:34.066450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.066024 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" event={"ID":"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443","Type":"ContainerDied","Data":"4ccd3099c633cbf55668cf9097130eea9ad521150e62b07873250a9bf3e0520a"} Nov 03 00:08:34.069084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.069053 5144 generic.go:356] "Generic (PLEG): container finished" podID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerID="e72fa2f645a3d6ef68b0d30bffcf39e62667a0033247fa312c7c28213785f383" exitCode=0 Nov 03 00:08:34.069232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.069198 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" event={"ID":"5c8d8690-756f-4734-98fa-9e4b1e610053","Type":"ContainerDied","Data":"e72fa2f645a3d6ef68b0d30bffcf39e62667a0033247fa312c7c28213785f383"} Nov 03 00:08:34.272205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.272174 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:34.434009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.433916 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf\" (UniqueName: \"kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf\") pod \"5e18ec5f-d608-413d-b3f7-610a1de01da1\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " Nov 03 00:08:34.434182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.434006 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle\") pod \"5e18ec5f-d608-413d-b3f7-610a1de01da1\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " Nov 03 00:08:34.434182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.434050 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util\") pod \"5e18ec5f-d608-413d-b3f7-610a1de01da1\" (UID: \"5e18ec5f-d608-413d-b3f7-610a1de01da1\") " Nov 03 00:08:34.434999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.434951 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle" pod "5e18ec5f-d608-413d-b3f7-610a1de01da1" (UID: "5e18ec5f-d608-413d-b3f7-610a1de01da1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:34.436895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.436867 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf" pod "5e18ec5f-d608-413d-b3f7-610a1de01da1" (UID: "5e18ec5f-d608-413d-b3f7-610a1de01da1"). InnerVolumeSpecName "kube-api-access-br4vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:34.440885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.440556 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util" pod "5e18ec5f-d608-413d-b3f7-610a1de01da1" (UID: "5e18ec5f-d608-413d-b3f7-610a1de01da1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:34.535588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.535497 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-br4vf\" (UniqueName: \"kubernetes.io/projected/5e18ec5f-d608-413d-b3f7-610a1de01da1-kube-api-access-br4vf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:34.535588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.535542 5144 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:34.535588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:34.535558 5144 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5e18ec5f-d608-413d-b3f7-610a1de01da1-util\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:35.079522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.079421 5144 generic.go:356] "Generic (PLEG): container finished" podID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerID="c1076cf2c5b4acc9d256faf9abc4abd35c0e2c5cf83ef7ce43b0964f79019339" exitCode=0 Nov 03 00:08:35.080040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.079519 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" event={"ID":"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443","Type":"ContainerDied","Data":"c1076cf2c5b4acc9d256faf9abc4abd35c0e2c5cf83ef7ce43b0964f79019339"} Nov 03 00:08:35.082394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.082371 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" Nov 03 00:08:35.082855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.082821 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be719w478" event={"ID":"5e18ec5f-d608-413d-b3f7-610a1de01da1","Type":"ContainerDied","Data":"84b8da9b953878a82e498455a72d03ffc794974764beb4aafa9a48c1ce89471d"} Nov 03 00:08:35.082997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.082876 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84b8da9b953878a82e498455a72d03ffc794974764beb4aafa9a48c1ce89471d" Nov 03 00:08:35.085777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.085748 5144 generic.go:356] "Generic (PLEG): container finished" podID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerID="e8eca194fc2406ec2925ffcd200faaf4df45de0efb4b84907a55f851614c544e" exitCode=0 Nov 03 00:08:35.085895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.085815 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" event={"ID":"6fd29826-fbc5-46e4-8333-045d9f7cced3","Type":"ContainerDied","Data":"e8eca194fc2406ec2925ffcd200faaf4df45de0efb4b84907a55f851614c544e"} Nov 03 00:08:35.234337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.234307 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:35.343942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.343858 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util\") pod \"5c8d8690-756f-4734-98fa-9e4b1e610053\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " Nov 03 00:08:35.343942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.343931 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle\") pod \"5c8d8690-756f-4734-98fa-9e4b1e610053\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " Nov 03 00:08:35.344173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.343989 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm\" (UniqueName: \"kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm\") pod \"5c8d8690-756f-4734-98fa-9e4b1e610053\" (UID: \"5c8d8690-756f-4734-98fa-9e4b1e610053\") " Nov 03 00:08:35.344543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.344508 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle" pod "5c8d8690-756f-4734-98fa-9e4b1e610053" (UID: "5c8d8690-756f-4734-98fa-9e4b1e610053"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:35.346605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.346571 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm" pod "5c8d8690-756f-4734-98fa-9e4b1e610053" (UID: "5c8d8690-756f-4734-98fa-9e4b1e610053"). InnerVolumeSpecName "kube-api-access-2k9jm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:35.348577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.348546 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util" pod "5c8d8690-756f-4734-98fa-9e4b1e610053" (UID: "5c8d8690-756f-4734-98fa-9e4b1e610053"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:35.445735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.445698 5144 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-util\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:35.445735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.445738 5144 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c8d8690-756f-4734-98fa-9e4b1e610053-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:35.445985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:35.445752 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k9jm\" (UniqueName: \"kubernetes.io/projected/5c8d8690-756f-4734-98fa-9e4b1e610053-kube-api-access-2k9jm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:36.094466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.093991 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" event={"ID":"07a95090-0b1a-4941-a1da-6cc79a48125f","Type":"ContainerStarted","Data":"16e763568e84922d4c01f42534c8217810d4668609a15afdf79f328b7ab5c170"} Nov 03 00:08:36.096793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.096763 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" Nov 03 00:08:36.096959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.096832 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4c6dthl" event={"ID":"5c8d8690-756f-4734-98fa-9e4b1e610053","Type":"ContainerDied","Data":"089ea798e18bd36958a05b999d54204d08f10e9081968eb7748d34c5666ea3e2"} Nov 03 00:08:36.096959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.096877 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="089ea798e18bd36958a05b999d54204d08f10e9081968eb7748d34c5666ea3e2" Nov 03 00:08:36.209665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.209500 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" podStartSLOduration=2.580873397 podStartE2EDuration="28.209479671s" totalImagesPullingTime="25.628606274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:08 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:36.208357865 +0000 UTC m=+206.354052347" watchObservedRunningTime="2026-11-03 00:08:36.209479671 +0000 UTC m=+206.355174129" Nov 03 00:08:36.255509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.255444 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:36.293417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.293390 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:36.352866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.352838 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m\" (UniqueName: \"kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m\") pod \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " Nov 03 00:08:36.355014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.352996 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle\") pod \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " Nov 03 00:08:36.355014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.353053 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util\") pod \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\" (UID: \"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443\") " Nov 03 00:08:36.355014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.353772 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle" pod "8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" (UID: "8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:36.355364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.355335 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m" pod "8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" (UID: "8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443"). InnerVolumeSpecName "kube-api-access-kwx8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:36.358576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.358548 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util" pod "8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" (UID: "8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:36.454525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.454483 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util\") pod \"6fd29826-fbc5-46e4-8333-045d9f7cced3\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " Nov 03 00:08:36.454714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.454535 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle\") pod \"6fd29826-fbc5-46e4-8333-045d9f7cced3\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " Nov 03 00:08:36.454714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.454608 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg\" (UniqueName: \"kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg\") pod \"6fd29826-fbc5-46e4-8333-045d9f7cced3\" (UID: \"6fd29826-fbc5-46e4-8333-045d9f7cced3\") " Nov 03 00:08:36.454869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.454854 5144 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:36.454901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.454874 5144 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-util\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:36.454901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.454888 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwx8m\" (UniqueName: \"kubernetes.io/projected/8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443-kube-api-access-kwx8m\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:36.455046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.455008 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle" pod "6fd29826-fbc5-46e4-8333-045d9f7cced3" (UID: "6fd29826-fbc5-46e4-8333-045d9f7cced3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:36.456876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.456851 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg" pod "6fd29826-fbc5-46e4-8333-045d9f7cced3" (UID: "6fd29826-fbc5-46e4-8333-045d9f7cced3"). InnerVolumeSpecName "kube-api-access-t58rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:36.460463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.460439 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util" pod "6fd29826-fbc5-46e4-8333-045d9f7cced3" (UID: "6fd29826-fbc5-46e4-8333-045d9f7cced3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:36.555605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.555570 5144 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-util\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:36.555605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.555602 5144 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fd29826-fbc5-46e4-8333-045d9f7cced3-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:36.555605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:36.555612 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t58rg\" (UniqueName: \"kubernetes.io/projected/6fd29826-fbc5-46e4-8333-045d9f7cced3-kube-api-access-t58rg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:37.103690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.103648 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" event={"ID":"86fa780e-92a2-4f70-ab20-41291309f56b","Type":"ContainerStarted","Data":"e331dfa6a3bc3ce6ddbb057b4e79b835cb87f80ea511a55beee0271b8663dae7"} Nov 03 00:08:37.105850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.105825 5144 generic.go:356] "Generic (PLEG): container finished" podID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerID="16e763568e84922d4c01f42534c8217810d4668609a15afdf79f328b7ab5c170" exitCode=0 Nov 03 00:08:37.105970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.105896 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" event={"ID":"07a95090-0b1a-4941-a1da-6cc79a48125f","Type":"ContainerDied","Data":"16e763568e84922d4c01f42534c8217810d4668609a15afdf79f328b7ab5c170"} Nov 03 00:08:37.107734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.107717 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" Nov 03 00:08:37.107819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.107769 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644rrtz4" event={"ID":"8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443","Type":"ContainerDied","Data":"143ca4fea6781d5448f2ef2bef279e1418d9fee71f4ec6ba55759bbd1d664960"} Nov 03 00:08:37.107819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.107811 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="143ca4fea6781d5448f2ef2bef279e1418d9fee71f4ec6ba55759bbd1d664960" Nov 03 00:08:37.109560 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.109536 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" event={"ID":"6fd29826-fbc5-46e4-8333-045d9f7cced3","Type":"ContainerDied","Data":"6eb4b7cc133fe07c5b76f8ad26f0bb93b992c08879689e2b955e4c07baa254bd"} Nov 03 00:08:37.109678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.109579 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6eb4b7cc133fe07c5b76f8ad26f0bb93b992c08879689e2b955e4c07baa254bd" Nov 03 00:08:37.109678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.109549 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7wcltw" Nov 03 00:08:37.219281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:37.219230 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" podStartSLOduration=3.203154035 podStartE2EDuration="28.219215152s" totalImagesPullingTime="25.016061117s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:09 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:37.218615776 +0000 UTC m=+207.364310249" watchObservedRunningTime="2026-11-03 00:08:37.219215152 +0000 UTC m=+207.364909648" Nov 03 00:08:38.118169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.118132 5144 generic.go:356] "Generic (PLEG): container finished" podID="86fa780e-92a2-4f70-ab20-41291309f56b" containerID="e331dfa6a3bc3ce6ddbb057b4e79b835cb87f80ea511a55beee0271b8663dae7" exitCode=0 Nov 03 00:08:38.118678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.118209 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" event={"ID":"86fa780e-92a2-4f70-ab20-41291309f56b","Type":"ContainerDied","Data":"e331dfa6a3bc3ce6ddbb057b4e79b835cb87f80ea511a55beee0271b8663dae7"} Nov 03 00:08:38.251602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.251570 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:38.271780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.271749 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util\") pod \"07a95090-0b1a-4941-a1da-6cc79a48125f\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " Nov 03 00:08:38.271914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.271813 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4\" (UniqueName: \"kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4\") pod \"07a95090-0b1a-4941-a1da-6cc79a48125f\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " Nov 03 00:08:38.271914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.271842 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle\") pod \"07a95090-0b1a-4941-a1da-6cc79a48125f\" (UID: \"07a95090-0b1a-4941-a1da-6cc79a48125f\") " Nov 03 00:08:38.272488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.272458 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle" pod "07a95090-0b1a-4941-a1da-6cc79a48125f" (UID: "07a95090-0b1a-4941-a1da-6cc79a48125f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:38.274015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.273989 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4" pod "07a95090-0b1a-4941-a1da-6cc79a48125f" (UID: "07a95090-0b1a-4941-a1da-6cc79a48125f"). InnerVolumeSpecName "kube-api-access-fcwp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:38.278024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.277988 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util" pod "07a95090-0b1a-4941-a1da-6cc79a48125f" (UID: "07a95090-0b1a-4941-a1da-6cc79a48125f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:38.372497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.372419 5144 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-util\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:38.372497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.372448 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcwp4\" (UniqueName: \"kubernetes.io/projected/07a95090-0b1a-4941-a1da-6cc79a48125f-kube-api-access-fcwp4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:38.372497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:38.372461 5144 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/07a95090-0b1a-4941-a1da-6cc79a48125f-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:39.124875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.124837 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" event={"ID":"07a95090-0b1a-4941-a1da-6cc79a48125f","Type":"ContainerDied","Data":"7e757d58897c86673e463e6aadbe3c9adba13ffd5687aa3efff13a145d4ebecc"} Nov 03 00:08:39.124875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.124868 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a99hm2" Nov 03 00:08:39.124875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.124890 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e757d58897c86673e463e6aadbe3c9adba13ffd5687aa3efff13a145d4ebecc" Nov 03 00:08:39.259867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.259841 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:39.279687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.279657 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t\" (UniqueName: \"kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t\") pod \"86fa780e-92a2-4f70-ab20-41291309f56b\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " Nov 03 00:08:39.279857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.279746 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util\") pod \"86fa780e-92a2-4f70-ab20-41291309f56b\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " Nov 03 00:08:39.279857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.279817 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle\") pod \"86fa780e-92a2-4f70-ab20-41291309f56b\" (UID: \"86fa780e-92a2-4f70-ab20-41291309f56b\") " Nov 03 00:08:39.280462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.280374 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle" pod "86fa780e-92a2-4f70-ab20-41291309f56b" (UID: "86fa780e-92a2-4f70-ab20-41291309f56b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:39.281961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.281933 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t" pod "86fa780e-92a2-4f70-ab20-41291309f56b" (UID: "86fa780e-92a2-4f70-ab20-41291309f56b"). InnerVolumeSpecName "kube-api-access-jrk2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:08:39.285025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.284999 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util" pod "86fa780e-92a2-4f70-ab20-41291309f56b" (UID: "86fa780e-92a2-4f70-ab20-41291309f56b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:08:39.381428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.381348 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrk2t\" (UniqueName: \"kubernetes.io/projected/86fa780e-92a2-4f70-ab20-41291309f56b-kube-api-access-jrk2t\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:39.381428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.381377 5144 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-util\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:39.381428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:39.381387 5144 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/86fa780e-92a2-4f70-ab20-41291309f56b-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:08:40.131837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.131797 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" event={"ID":"86fa780e-92a2-4f70-ab20-41291309f56b","Type":"ContainerDied","Data":"6f469191f2fb082ff5bc9117e56ca70c22bbac772df4d765ad271e38024673d5"} Nov 03 00:08:40.131837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.131841 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651chz567" Nov 03 00:08:40.132351 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.131846 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f469191f2fb082ff5bc9117e56ca70c22bbac772df4d765ad271e38024673d5" Nov 03 00:08:40.524857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.524821 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5"] Nov 03 00:08:40.525143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525131 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="extract" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525145 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="extract" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525160 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="pull" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525165 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="pull" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525173 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="pull" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525178 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="pull" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525184 5144 cpu_manager.go:414] "containerMap: removing container" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="pull" Nov 03 00:08:40.525188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525188 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525195 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525201 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525212 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525217 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525228 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525232 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525237 5144 cpu_manager.go:414] "containerMap: removing container" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525242 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525249 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525253 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525259 5144 cpu_manager.go:414] "containerMap: removing container" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525263 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525268 5144 cpu_manager.go:414] "containerMap: removing container" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525272 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525284 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525289 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525294 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525298 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525304 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525308 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="pull" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525314 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525319 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525325 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525329 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525338 5144 cpu_manager.go:414] "containerMap: removing container" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525342 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="util" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525348 5144 cpu_manager.go:414] "containerMap: removing container" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525352 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="extract" Nov 03 00:08:40.525385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525399 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="07a95090-0b1a-4941-a1da-6cc79a48125f" containerName="extract" Nov 03 00:08:40.526213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525406 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd29826-fbc5-46e4-8333-045d9f7cced3" containerName="extract" Nov 03 00:08:40.526213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525412 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e18ec5f-d608-413d-b3f7-610a1de01da1" containerName="extract" Nov 03 00:08:40.526213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525419 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c8d8690-756f-4734-98fa-9e4b1e610053" containerName="extract" Nov 03 00:08:40.526213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525425 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="86fa780e-92a2-4f70-ab20-41291309f56b" containerName="extract" Nov 03 00:08:40.526213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.525431 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f8c8aa1-8cd3-4aa4-b306-d27ccdab5443" containerName="extract" Nov 03 00:08:40.527744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.527727 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.580085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.580054 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 00:08:40.580085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.580054 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 00:08:40.580085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.580067 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-9xr6v\"" Nov 03 00:08:40.580318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.580067 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 00:08:40.580318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.580091 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 00:08:40.580318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.580219 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 00:08:40.589575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.589549 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c233239-62f4-444d-9654-a09cc00da2fc-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.589719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.589576 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.589719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.589600 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.589719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.589619 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c233239-62f4-444d-9654-a09cc00da2fc-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.589883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.589794 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.589883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.589835 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s68j\" (UniqueName: \"kubernetes.io/projected/9c233239-62f4-444d-9654-a09cc00da2fc-kube-api-access-9s68j\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690376 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c233239-62f4-444d-9654-a09cc00da2fc-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690406 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690447 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690471 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c233239-62f4-444d-9654-a09cc00da2fc-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690509 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690527 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s68j\" (UniqueName: \"kubernetes.io/projected/9c233239-62f4-444d-9654-a09cc00da2fc-kube-api-access-9s68j\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.690839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.690799 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/9c233239-62f4-444d-9654-a09cc00da2fc-manifests\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.693043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.693010 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9c233239-62f4-444d-9654-a09cc00da2fc-tmp\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.693174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.693115 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-webhook-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.693237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.693219 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.693298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.693235 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9c233239-62f4-444d-9654-a09cc00da2fc-metrics-cert\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.753293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.753256 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s68j\" (UniqueName: \"kubernetes.io/projected/9c233239-62f4-444d-9654-a09cc00da2fc-kube-api-access-9s68j\") pod \"opendatahub-operator-controller-manager-65f97f758d-h9cj5\" (UID: \"9c233239-62f4-444d-9654-a09cc00da2fc\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:40.765771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.765741 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5"] Nov 03 00:08:40.840511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:40.840409 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:41.375158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:41.375082 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5"] Nov 03 00:08:41.377453 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:41.377428 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c233239_62f4_444d_9654_a09cc00da2fc.slice/crio-8358f4a667244eeb7b6bb134bd0bd375fcc0a9fd391f1cc62acfc09e9322fa2b WatchSource:0}: Error finding container 8358f4a667244eeb7b6bb134bd0bd375fcc0a9fd391f1cc62acfc09e9322fa2b: Status 404 returned error can't find the container with id 8358f4a667244eeb7b6bb134bd0bd375fcc0a9fd391f1cc62acfc09e9322fa2b Nov 03 00:08:42.143210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:42.143169 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" event={"ID":"9c233239-62f4-444d-9654-a09cc00da2fc","Type":"ContainerStarted","Data":"8358f4a667244eeb7b6bb134bd0bd375fcc0a9fd391f1cc62acfc09e9322fa2b"} Nov 03 00:08:42.155278 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:08:42.155230 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:08:45.166324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:45.166287 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" event={"ID":"9c233239-62f4-444d-9654-a09cc00da2fc","Type":"ContainerStarted","Data":"745e8a07f999f07435fdfb5e137145fd3f1be059e057ffc7a51d30703c1241c9"} Nov 03 00:08:46.172892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:46.172855 5144 generic.go:356] "Generic (PLEG): container finished" podID="9c233239-62f4-444d-9654-a09cc00da2fc" containerID="745e8a07f999f07435fdfb5e137145fd3f1be059e057ffc7a51d30703c1241c9" exitCode=0 Nov 03 00:08:46.173284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:46.172936 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" event={"ID":"9c233239-62f4-444d-9654-a09cc00da2fc","Type":"ContainerDied","Data":"745e8a07f999f07435fdfb5e137145fd3f1be059e057ffc7a51d30703c1241c9"} Nov 03 00:08:48.183825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:48.183791 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" event={"ID":"9c233239-62f4-444d-9654-a09cc00da2fc","Type":"ContainerStarted","Data":"88ec05e6ca11f04b4d4b6f66a73af527507dace957386bda47fa9927d049973a"} Nov 03 00:08:48.184284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:48.183914 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:48.466165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:48.466035 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" podStartSLOduration=3.63832719 podStartE2EDuration="8.466014789s" totalImagesPullingTime="4.326514717s" totalInitContainerRuntime="501.172882ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:48.415144012 +0000 UTC m=+218.560838487" watchObservedRunningTime="2026-11-03 00:08:48.466014789 +0000 UTC m=+218.611709250" Nov 03 00:08:58.958234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:58.958200 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n"] Nov 03 00:08:59.058380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.058339 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n"] Nov 03 00:08:59.058595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.058451 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.110985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.110946 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hsddz\"" Nov 03 00:08:59.110985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.110949 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Nov 03 00:08:59.111194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.110964 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Nov 03 00:08:59.148058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.148014 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spz4v\" (UniqueName: \"kubernetes.io/projected/2756e826-7114-4699-9d66-50cd2777e028-kube-api-access-spz4v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ng57n\" (UID: \"2756e826-7114-4699-9d66-50cd2777e028\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.148246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.148152 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2756e826-7114-4699-9d66-50cd2777e028-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ng57n\" (UID: \"2756e826-7114-4699-9d66-50cd2777e028\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.190853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.190823 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-65f97f758d-h9cj5" Nov 03 00:08:59.248708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.248580 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2756e826-7114-4699-9d66-50cd2777e028-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ng57n\" (UID: \"2756e826-7114-4699-9d66-50cd2777e028\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.248881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.248818 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spz4v\" (UniqueName: \"kubernetes.io/projected/2756e826-7114-4699-9d66-50cd2777e028-kube-api-access-spz4v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ng57n\" (UID: \"2756e826-7114-4699-9d66-50cd2777e028\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.249024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.249001 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2756e826-7114-4699-9d66-50cd2777e028-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ng57n\" (UID: \"2756e826-7114-4699-9d66-50cd2777e028\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.305403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.305362 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spz4v\" (UniqueName: \"kubernetes.io/projected/2756e826-7114-4699-9d66-50cd2777e028-kube-api-access-spz4v\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-ng57n\" (UID: \"2756e826-7114-4699-9d66-50cd2777e028\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.395275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.395238 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" Nov 03 00:08:59.861050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:08:59.861028 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n"] Nov 03 00:08:59.863422 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:08:59.863383 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2756e826_7114_4699_9d66_50cd2777e028.slice/crio-0851c7d696e190887559e3f4874b7d77e14c110436988b3dc06f2b113e6e327d WatchSource:0}: Error finding container 0851c7d696e190887559e3f4874b7d77e14c110436988b3dc06f2b113e6e327d: Status 404 returned error can't find the container with id 0851c7d696e190887559e3f4874b7d77e14c110436988b3dc06f2b113e6e327d Nov 03 00:09:00.241197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:00.241103 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" event={"ID":"2756e826-7114-4699-9d66-50cd2777e028","Type":"ContainerStarted","Data":"0851c7d696e190887559e3f4874b7d77e14c110436988b3dc06f2b113e6e327d"} Nov 03 00:09:01.426360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.426275 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5"] Nov 03 00:09:01.477744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.477708 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.482579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.482547 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5"] Nov 03 00:09:01.542284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.542248 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 00:09:01.542284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.542255 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 00:09:01.572733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.572693 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjpb7\" (UniqueName: \"kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7\") pod \"kuadrant-operator-controller-manager-678d87766f-wkpw5\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.572924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.572852 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wkpw5\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.599194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.599166 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5fwfj\"" Nov 03 00:09:01.673471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.673430 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjpb7\" (UniqueName: \"kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7\") pod \"kuadrant-operator-controller-manager-678d87766f-wkpw5\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.673725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.673533 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wkpw5\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.673972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.673948 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-wkpw5\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.750807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.750733 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjpb7\" (UniqueName: \"kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7\") pod \"kuadrant-operator-controller-manager-678d87766f-wkpw5\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:01.809740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:01.809698 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:02.309903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:02.309828 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5"] Nov 03 00:09:02.312828 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:09:02.312798 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb2b0f0_fb57_4948_b3a2_4d9094b4992c.slice/crio-652937fdddaf35d8f8d7d4a50709a3b20a14bdd0c4d3aa1ff1b351a7c44da2c4 WatchSource:0}: Error finding container 652937fdddaf35d8f8d7d4a50709a3b20a14bdd0c4d3aa1ff1b351a7c44da2c4: Status 404 returned error can't find the container with id 652937fdddaf35d8f8d7d4a50709a3b20a14bdd0c4d3aa1ff1b351a7c44da2c4 Nov 03 00:09:03.256600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:03.256548 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" event={"ID":"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c","Type":"ContainerStarted","Data":"652937fdddaf35d8f8d7d4a50709a3b20a14bdd0c4d3aa1ff1b351a7c44da2c4"} Nov 03 00:09:05.267070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:05.267030 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" event={"ID":"2756e826-7114-4699-9d66-50cd2777e028","Type":"ContainerStarted","Data":"415b549a0e38b0407ea4a4dd3cf71aa3ea66dd97cf27f356806271033b8e5371"} Nov 03 00:09:05.488070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:05.488013 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-ng57n" podStartSLOduration=3.013378249 podStartE2EDuration="7.487998754s" totalImagesPullingTime="4.474620505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:05.486668747 +0000 UTC m=+235.632363232" watchObservedRunningTime="2026-11-03 00:09:05.487998754 +0000 UTC m=+235.633693208" Nov 03 00:09:09.081984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.081945 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4"] Nov 03 00:09:09.084214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.084196 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.138679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.138611 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Nov 03 00:09:09.141184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.140995 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phf77\" (UniqueName: \"kubernetes.io/projected/f2853041-c404-4322-b265-6e3bf7e76235-kube-api-access-phf77\") pod \"cert-manager-cainjector-5c57bbd4fc-67hq4\" (UID: \"f2853041-c404-4322-b265-6e3bf7e76235\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.141184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.141054 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2853041-c404-4322-b265-6e3bf7e76235-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-67hq4\" (UID: \"f2853041-c404-4322-b265-6e3bf7e76235\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.188660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.188593 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-gg488\"" Nov 03 00:09:09.190331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.190302 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4"] Nov 03 00:09:09.238222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.238187 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Nov 03 00:09:09.241509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.241488 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2853041-c404-4322-b265-6e3bf7e76235-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-67hq4\" (UID: \"f2853041-c404-4322-b265-6e3bf7e76235\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.241586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.241561 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phf77\" (UniqueName: \"kubernetes.io/projected/f2853041-c404-4322-b265-6e3bf7e76235-kube-api-access-phf77\") pod \"cert-manager-cainjector-5c57bbd4fc-67hq4\" (UID: \"f2853041-c404-4322-b265-6e3bf7e76235\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.297810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.297774 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phf77\" (UniqueName: \"kubernetes.io/projected/f2853041-c404-4322-b265-6e3bf7e76235-kube-api-access-phf77\") pod \"cert-manager-cainjector-5c57bbd4fc-67hq4\" (UID: \"f2853041-c404-4322-b265-6e3bf7e76235\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.298089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.298028 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f2853041-c404-4322-b265-6e3bf7e76235-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-67hq4\" (UID: \"f2853041-c404-4322-b265-6e3bf7e76235\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.408820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.408739 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" Nov 03 00:09:09.789240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:09.789205 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4"] Nov 03 00:09:09.793526 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:09:09.793498 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf2853041_c404_4322_b265_6e3bf7e76235.slice/crio-13c726331f690c00b5816925797b172102a212e547bc1b4918ca0e51a934fb0d WatchSource:0}: Error finding container 13c726331f690c00b5816925797b172102a212e547bc1b4918ca0e51a934fb0d: Status 404 returned error can't find the container with id 13c726331f690c00b5816925797b172102a212e547bc1b4918ca0e51a934fb0d Nov 03 00:09:10.297960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:10.297926 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" event={"ID":"f2853041-c404-4322-b265-6e3bf7e76235","Type":"ContainerStarted","Data":"13c726331f690c00b5816925797b172102a212e547bc1b4918ca0e51a934fb0d"} Nov 03 00:09:12.160160 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:09:12.160116 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:09:13.320938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:13.320894 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" event={"ID":"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c","Type":"ContainerStarted","Data":"5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5"} Nov 03 00:09:13.321438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:13.321050 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:13.452358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:13.452253 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" podStartSLOduration=1.5961151949999999 podStartE2EDuration="12.452235937s" totalImagesPullingTime="10.856120742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:13.450381305 +0000 UTC m=+243.596075800" watchObservedRunningTime="2026-11-03 00:09:13.452235937 +0000 UTC m=+243.597930393" Nov 03 00:09:16.337773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:16.337738 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" event={"ID":"f2853041-c404-4322-b265-6e3bf7e76235","Type":"ContainerStarted","Data":"e7f633c81eb608dada9718c51736f53294ea487e6e11cb8eca5bc47e83cd3938"} Nov 03 00:09:16.450148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:16.450080 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-67hq4" podStartSLOduration=1.439312006 podStartE2EDuration="7.450061889s" totalImagesPullingTime="6.010749883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:16.449123115 +0000 UTC m=+246.594817571" watchObservedRunningTime="2026-11-03 00:09:16.450061889 +0000 UTC m=+246.595756346" Nov 03 00:09:20.605704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.605657 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54744779bc-6nckb"] Nov 03 00:09:20.607941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.607918 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.664858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.664820 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 00:09:20.679126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.679093 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 00:09:20.709779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.709740 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k9nfq\"" Nov 03 00:09:20.711987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.711956 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54744779bc-6nckb"] Nov 03 00:09:20.747051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.746991 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-tmp\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.747312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.747153 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-tls-certs\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.747312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.747187 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8mk4\" (UniqueName: \"kubernetes.io/projected/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-kube-api-access-k8mk4\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.848397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.848360 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-tls-certs\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.848397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.848405 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8mk4\" (UniqueName: \"kubernetes.io/projected/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-kube-api-access-k8mk4\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.848685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.848439 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-tmp\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.850986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.850964 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-tmp\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.851199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.851178 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-tls-certs\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.908459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.908385 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8mk4\" (UniqueName: \"kubernetes.io/projected/9b8e9601-f07e-4271-8d87-a6bfbbe91d9d-kube-api-access-k8mk4\") pod \"kube-auth-proxy-54744779bc-6nckb\" (UID: \"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d\") " pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:20.919234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:20.919205 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" Nov 03 00:09:21.107433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.107396 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj"] Nov 03 00:09:21.109812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.109795 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.164029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.163933 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-6mncd\"" Nov 03 00:09:21.170264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.170233 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj"] Nov 03 00:09:21.252910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.252875 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.252910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.252912 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a5e11-095c-402f-bfeb-07032cc4455c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253003 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253048 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253094 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7t6q\" (UniqueName: \"kubernetes.io/projected/e01a5e11-095c-402f-bfeb-07032cc4455c-kube-api-access-z7t6q\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253139 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253173 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253196 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.253312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.253218 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.293332 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.293179 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54744779bc-6nckb"] Nov 03 00:09:21.295582 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:09:21.295554 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8e9601_f07e_4271_8d87_a6bfbbe91d9d.slice/crio-64d0b8449ccaeae4038b800c5c42fa9a7c25170471b9c56adaf302ede2c1345a WatchSource:0}: Error finding container 64d0b8449ccaeae4038b800c5c42fa9a7c25170471b9c56adaf302ede2c1345a: Status 404 returned error can't find the container with id 64d0b8449ccaeae4038b800c5c42fa9a7c25170471b9c56adaf302ede2c1345a Nov 03 00:09:21.354487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354442 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.354719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354520 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.354719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354556 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.354719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354594 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.354719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354673 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.354719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354705 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a5e11-095c-402f-bfeb-07032cc4455c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354764 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354798 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354828 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7t6q\" (UniqueName: \"kubernetes.io/projected/e01a5e11-095c-402f-bfeb-07032cc4455c-kube-api-access-z7t6q\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354882 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-data\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.354952 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.355346 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.355550 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e01a5e11-095c-402f-bfeb-07032cc4455c-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.355697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.355625 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.357442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.357407 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.357914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.357894 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.361431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.361402 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" event={"ID":"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d","Type":"ContainerStarted","Data":"64d0b8449ccaeae4038b800c5c42fa9a7c25170471b9c56adaf302ede2c1345a"} Nov 03 00:09:21.413944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.413907 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7t6q\" (UniqueName: \"kubernetes.io/projected/e01a5e11-095c-402f-bfeb-07032cc4455c-kube-api-access-z7t6q\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.416612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.416544 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e01a5e11-095c-402f-bfeb-07032cc4455c-istio-token\") pod \"data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj\" (UID: \"e01a5e11-095c-402f-bfeb-07032cc4455c\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.421379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.421340 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:21.847302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.847281 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj"] Nov 03 00:09:21.849694 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:09:21.849666 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01a5e11_095c_402f_bfeb_07032cc4455c.slice/crio-9c17de22efd9e4db68be783d8de9bcb8feb0d6c3044b1fc0344d7eb9fd858135 WatchSource:0}: Error finding container 9c17de22efd9e4db68be783d8de9bcb8feb0d6c3044b1fc0344d7eb9fd858135: Status 404 returned error can't find the container with id 9c17de22efd9e4db68be783d8de9bcb8feb0d6c3044b1fc0344d7eb9fd858135 Nov 03 00:09:21.852107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.852070 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:09:21.852220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.852150 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:09:21.852284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:21.852224 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:09:22.367967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:22.367930 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" event={"ID":"e01a5e11-095c-402f-bfeb-07032cc4455c","Type":"ContainerStarted","Data":"6444b8f2458c74566a96ca6f74bb02f459b3744216a47f26802f0f2ae0c8adb5"} Nov 03 00:09:22.368138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:22.367981 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" event={"ID":"e01a5e11-095c-402f-bfeb-07032cc4455c","Type":"ContainerStarted","Data":"9c17de22efd9e4db68be783d8de9bcb8feb0d6c3044b1fc0344d7eb9fd858135"} Nov 03 00:09:22.422113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:22.422085 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:22.427240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:22.427217 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:22.484545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:22.484487 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" podStartSLOduration=1.484469029 podStartE2EDuration="1.484469029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:22.483137425 +0000 UTC m=+252.628831879" watchObservedRunningTime="2026-11-03 00:09:22.484469029 +0000 UTC m=+252.630163488" Nov 03 00:09:23.373599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:23.373561 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:23.374660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:23.374609 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj" Nov 03 00:09:24.328588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:24.328559 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 00:09:24.381534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:24.381500 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" event={"ID":"9b8e9601-f07e-4271-8d87-a6bfbbe91d9d","Type":"ContainerStarted","Data":"48e922709e843a3537df34f2e6c0a30cc3bb70ac8010d72ead894ab463eeebb4"} Nov 03 00:09:24.604460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:24.604393 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54744779bc-6nckb" podStartSLOduration=1.689913181 podStartE2EDuration="4.604370341s" totalImagesPullingTime="2.91445716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:24.603979493 +0000 UTC m=+254.749673966" watchObservedRunningTime="2026-11-03 00:09:24.604370341 +0000 UTC m=+254.750064799" Nov 03 00:09:42.155957 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:09:42.155916 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:09:46.310868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.310834 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6"] Nov 03 00:09:46.313310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.313292 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.370190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.370157 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6"] Nov 03 00:09:46.374993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.374969 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6q47\" (UniqueName: \"kubernetes.io/projected/57f5d573-36e8-4eed-a770-e43e96ee61b4-kube-api-access-j6q47\") pod \"postgres-7cdbf6dc44-pgff6\" (UID: \"57f5d573-36e8-4eed-a770-e43e96ee61b4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.375136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.375013 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57f5d573-36e8-4eed-a770-e43e96ee61b4-data\") pod \"postgres-7cdbf6dc44-pgff6\" (UID: \"57f5d573-36e8-4eed-a770-e43e96ee61b4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.413261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.413229 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 00:09:46.413401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.413313 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 00:09:46.417001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.416987 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 00:09:46.417069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.417037 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-kcvws\"" Nov 03 00:09:46.475910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.475881 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57f5d573-36e8-4eed-a770-e43e96ee61b4-data\") pod \"postgres-7cdbf6dc44-pgff6\" (UID: \"57f5d573-36e8-4eed-a770-e43e96ee61b4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.476090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.475973 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6q47\" (UniqueName: \"kubernetes.io/projected/57f5d573-36e8-4eed-a770-e43e96ee61b4-kube-api-access-j6q47\") pod \"postgres-7cdbf6dc44-pgff6\" (UID: \"57f5d573-36e8-4eed-a770-e43e96ee61b4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.476254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.476236 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/57f5d573-36e8-4eed-a770-e43e96ee61b4-data\") pod \"postgres-7cdbf6dc44-pgff6\" (UID: \"57f5d573-36e8-4eed-a770-e43e96ee61b4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.535297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.535272 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6q47\" (UniqueName: \"kubernetes.io/projected/57f5d573-36e8-4eed-a770-e43e96ee61b4-kube-api-access-j6q47\") pod \"postgres-7cdbf6dc44-pgff6\" (UID: \"57f5d573-36e8-4eed-a770-e43e96ee61b4\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.624380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.624311 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:46.992624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:46.992597 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6"] Nov 03 00:09:46.994865 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:09:46.994837 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57f5d573_36e8_4eed_a770_e43e96ee61b4.slice/crio-dd80db4c1c387a21c2431177b1f57a97963a7e50babb777cb8bfd41d0a6a31c9 WatchSource:0}: Error finding container dd80db4c1c387a21c2431177b1f57a97963a7e50babb777cb8bfd41d0a6a31c9: Status 404 returned error can't find the container with id dd80db4c1c387a21c2431177b1f57a97963a7e50babb777cb8bfd41d0a6a31c9 Nov 03 00:09:47.497777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:47.497736 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" event={"ID":"57f5d573-36e8-4eed-a770-e43e96ee61b4","Type":"ContainerStarted","Data":"dd80db4c1c387a21c2431177b1f57a97963a7e50babb777cb8bfd41d0a6a31c9"} Nov 03 00:09:54.535365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:54.535331 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" event={"ID":"57f5d573-36e8-4eed-a770-e43e96ee61b4","Type":"ContainerStarted","Data":"ed6fad41fb4bacb4032da8de9ad75629da16452f4eddb7f855084491d1f4a50a"} Nov 03 00:09:54.535827 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:54.535499 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:09:54.650149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:09:54.650100 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" podStartSLOduration=2.068950342 podStartE2EDuration="8.650087575s" totalImagesPullingTime="6.581137233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:09:54.648706729 +0000 UTC m=+284.794401203" watchObservedRunningTime="2026-11-03 00:09:54.650087575 +0000 UTC m=+284.795782030" Nov 03 00:10:00.570601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:00.570573 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-pgff6" Nov 03 00:10:10.022650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:10.022603 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:10:10.023910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:10.023888 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:10:12.155307 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:10:12.155266 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:10:15.225231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.225197 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c"] Nov 03 00:10:15.231026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.231000 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.285536 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.285503 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c"] Nov 03 00:10:15.333024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.332990 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Nov 03 00:10:15.333024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.333012 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 00:10:15.333024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.332999 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Nov 03 00:10:15.333318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.333029 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Nov 03 00:10:15.333318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.333161 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 00:10:15.333531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.333500 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-xsk4r\"" Nov 03 00:10:15.333963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.333939 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b5vc\" (UniqueName: \"kubernetes.io/projected/916b609e-7a1f-4137-83b5-424605bc72b0-kube-api-access-6b5vc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.334059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.334007 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.334125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.334108 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.334173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.334159 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916b609e-7a1f-4137-83b5-424605bc72b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.434825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.434786 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.435002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.434833 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916b609e-7a1f-4137-83b5-424605bc72b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.435002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.434879 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b5vc\" (UniqueName: \"kubernetes.io/projected/916b609e-7a1f-4137-83b5-424605bc72b0-kube-api-access-6b5vc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.435002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.434914 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.435002 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:10:15.434918 5144 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found Nov 03 00:10:15.435148 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:10:15.435012 5144 secret.go:189] Couldn't get secret opendatahub/usage-logs-tenancy-proxy-tls: secret "usage-logs-tenancy-proxy-tls" not found Nov 03 00:10:15.435148 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:10:15.435024 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-proxy-source podName:916b609e-7a1f-4137-83b5-424605bc72b0 nodeName:}" failed. No retries permitted until 2026-11-03 00:10:15.935000596 +0000 UTC m=+306.080695049 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" (UID: "916b609e-7a1f-4137-83b5-424605bc72b0") : configmap "usage-tenancy-proxy-source" not found Nov 03 00:10:15.435148 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:10:15.435049 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/916b609e-7a1f-4137-83b5-424605bc72b0-tls-cert podName:916b609e-7a1f-4137-83b5-424605bc72b0 nodeName:}" failed. No retries permitted until 2026-11-03 00:10:15.935039266 +0000 UTC m=+306.080733706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-cert" (UniqueName: "kubernetes.io/secret/916b609e-7a1f-4137-83b5-424605bc72b0-tls-cert") pod "usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" (UID: "916b609e-7a1f-4137-83b5-424605bc72b0") : secret "usage-logs-tenancy-proxy-tls" not found Nov 03 00:10:15.435534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.435517 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.493471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.493404 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b5vc\" (UniqueName: \"kubernetes.io/projected/916b609e-7a1f-4137-83b5-424605bc72b0-kube-api-access-6b5vc\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.938984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.938951 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916b609e-7a1f-4137-83b5-424605bc72b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.939175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.939069 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.939822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.939804 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/916b609e-7a1f-4137-83b5-424605bc72b0-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:15.941772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:15.941748 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/916b609e-7a1f-4137-83b5-424605bc72b0-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-m2z5c\" (UID: \"916b609e-7a1f-4137-83b5-424605bc72b0\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:16.145280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:16.145253 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:16.526210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:16.526188 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c"] Nov 03 00:10:16.528463 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:10:16.528432 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916b609e_7a1f_4137_83b5_424605bc72b0.slice/crio-7315284f4507756ab0b3fcdc037f885efea36e9df7fe2356ed49e980716d9321 WatchSource:0}: Error finding container 7315284f4507756ab0b3fcdc037f885efea36e9df7fe2356ed49e980716d9321: Status 404 returned error can't find the container with id 7315284f4507756ab0b3fcdc037f885efea36e9df7fe2356ed49e980716d9321 Nov 03 00:10:16.530480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:16.530455 5144 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:10:16.640911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:16.640874 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" event={"ID":"916b609e-7a1f-4137-83b5-424605bc72b0","Type":"ContainerStarted","Data":"7315284f4507756ab0b3fcdc037f885efea36e9df7fe2356ed49e980716d9321"} Nov 03 00:10:33.740300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:33.740261 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" event={"ID":"916b609e-7a1f-4137-83b5-424605bc72b0","Type":"ContainerStarted","Data":"0376407d0fef97965d16fe565a5c4cd048a3c2939497dfe972c23a2af58dedfe"} Nov 03 00:10:33.740793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:33.740362 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:33.857216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:33.857162 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" podStartSLOduration=2.533063345 podStartE2EDuration="18.857146946s" totalImagesPullingTime="16.324083601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:33.855758442 +0000 UTC m=+324.001452964" watchObservedRunningTime="2026-11-03 00:10:33.857146946 +0000 UTC m=+324.002841402" Nov 03 00:10:39.256815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.256771 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8q225"] Nov 03 00:10:39.260798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.260773 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.364325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.364286 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-rpz65\"" Nov 03 00:10:39.364325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.364288 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 00:10:39.366260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.366233 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8q225"] Nov 03 00:10:39.374775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.374749 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/2dc160d5-869c-4f83-9abc-0f3a0018740f-kube-api-access-d945d\") pod \"model-serving-api-75bd89fdfc-8q225\" (UID: \"2dc160d5-869c-4f83-9abc-0f3a0018740f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.374899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.374877 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc160d5-869c-4f83-9abc-0f3a0018740f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8q225\" (UID: \"2dc160d5-869c-4f83-9abc-0f3a0018740f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.475400 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.475362 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc160d5-869c-4f83-9abc-0f3a0018740f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8q225\" (UID: \"2dc160d5-869c-4f83-9abc-0f3a0018740f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.475599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.475423 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/2dc160d5-869c-4f83-9abc-0f3a0018740f-kube-api-access-d945d\") pod \"model-serving-api-75bd89fdfc-8q225\" (UID: \"2dc160d5-869c-4f83-9abc-0f3a0018740f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.477977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.477954 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dc160d5-869c-4f83-9abc-0f3a0018740f-tls-certs\") pod \"model-serving-api-75bd89fdfc-8q225\" (UID: \"2dc160d5-869c-4f83-9abc-0f3a0018740f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.537209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.537138 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d945d\" (UniqueName: \"kubernetes.io/projected/2dc160d5-869c-4f83-9abc-0f3a0018740f-kube-api-access-d945d\") pod \"model-serving-api-75bd89fdfc-8q225\" (UID: \"2dc160d5-869c-4f83-9abc-0f3a0018740f\") " pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.574154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.574125 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:39.946704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:39.946676 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-8q225"] Nov 03 00:10:39.948206 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:10:39.948178 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc160d5_869c_4f83_9abc_0f3a0018740f.slice/crio-29de7933a076fb2ec7f1fd40a301bab83b36586642721aec7cf45f2ceaf5aaca WatchSource:0}: Error finding container 29de7933a076fb2ec7f1fd40a301bab83b36586642721aec7cf45f2ceaf5aaca: Status 404 returned error can't find the container with id 29de7933a076fb2ec7f1fd40a301bab83b36586642721aec7cf45f2ceaf5aaca Nov 03 00:10:40.774381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:40.774341 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" event={"ID":"2dc160d5-869c-4f83-9abc-0f3a0018740f","Type":"ContainerStarted","Data":"29de7933a076fb2ec7f1fd40a301bab83b36586642721aec7cf45f2ceaf5aaca"} Nov 03 00:10:42.156977 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:10:42.156943 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:10:42.786282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:42.786244 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" event={"ID":"2dc160d5-869c-4f83-9abc-0f3a0018740f","Type":"ContainerStarted","Data":"c49cb26ce568dee0ea40e4a4b22eee522e9ae3b9f8e0031d6fd34e7189492888"} Nov 03 00:10:42.786469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:42.786340 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:10:42.901426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:42.901372 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" podStartSLOduration=1.378598536 podStartE2EDuration="3.901351706s" totalImagesPullingTime="2.52275317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:42.899534034 +0000 UTC m=+333.045228517" watchObservedRunningTime="2026-11-03 00:10:42.901351706 +0000 UTC m=+333.047046162" Nov 03 00:10:44.751302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:44.751265 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-m2z5c" Nov 03 00:10:53.794862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:10:53.794825 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-8q225" Nov 03 00:11:03.397090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.397050 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzv6h"] Nov 03 00:11:03.420174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.420145 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:03.453405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.453371 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzv6h"] Nov 03 00:11:03.501041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.501010 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7ns8x\"" Nov 03 00:11:03.527416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.527376 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh45t\" (UniqueName: \"kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t\") pod \"authorino-6f6944bcdd-wzv6h\" (UID: \"99b6d6fe-c32c-4445-b7ec-b35924e3a46e\") " pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:03.628477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.628434 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh45t\" (UniqueName: \"kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t\") pod \"authorino-6f6944bcdd-wzv6h\" (UID: \"99b6d6fe-c32c-4445-b7ec-b35924e3a46e\") " pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:03.684938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.684856 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh45t\" (UniqueName: \"kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t\") pod \"authorino-6f6944bcdd-wzv6h\" (UID: \"99b6d6fe-c32c-4445-b7ec-b35924e3a46e\") " pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:03.735559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.735519 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzv6h"] Nov 03 00:11:03.735778 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.735753 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:03.896360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.896330 5144 generic.go:356] "Generic (PLEG): container finished" podID="281353eb-04c5-4896-bf1e-b6948829ac54" containerID="2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c" exitCode=0 Nov 03 00:11:03.896529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.896411 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" event={"ID":"281353eb-04c5-4896-bf1e-b6948829ac54","Type":"ContainerDied","Data":"2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c"} Nov 03 00:11:03.896867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.896851 5144 scope.go:122] "RemoveContainer" containerID="2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c" Nov 03 00:11:03.897989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.897957 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:11:03.898085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.898031 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:11:03.898085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:03.898064 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:11:04.126317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:04.126291 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzv6h"] Nov 03 00:11:04.128142 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:11:04.128115 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99b6d6fe_c32c_4445_b7ec_b35924e3a46e.slice/crio-bb16b21099c1b7dabfa115177eb7e305796798729d3f3043041c4acb9b89e2f3 WatchSource:0}: Error finding container bb16b21099c1b7dabfa115177eb7e305796798729d3f3043041c4acb9b89e2f3: Status 404 returned error can't find the container with id bb16b21099c1b7dabfa115177eb7e305796798729d3f3043041c4acb9b89e2f3 Nov 03 00:11:04.902735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:04.902696 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" event={"ID":"99b6d6fe-c32c-4445-b7ec-b35924e3a46e","Type":"ContainerStarted","Data":"bb16b21099c1b7dabfa115177eb7e305796798729d3f3043041c4acb9b89e2f3"} Nov 03 00:11:04.904859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:04.904832 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" event={"ID":"281353eb-04c5-4896-bf1e-b6948829ac54","Type":"ContainerStarted","Data":"8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8"} Nov 03 00:11:05.040597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:05.040562 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:11:05.040878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:05.040853 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 00:11:05.040940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:05.040904 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 00:11:06.041334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:06.041295 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 00:11:06.041822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:06.041368 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 00:11:07.041204 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:07.041169 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 00:11:07.041446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:07.041418 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 00:11:07.950121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:07.950069 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" event={"ID":"99b6d6fe-c32c-4445-b7ec-b35924e3a46e","Type":"ContainerStarted","Data":"4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9"} Nov 03 00:11:07.950121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:07.950111 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" podUID="99b6d6fe-c32c-4445-b7ec-b35924e3a46e" containerName="authorino" containerID="cri-o://4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9" gracePeriod=30 Nov 03 00:11:08.040988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.040946 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 00:11:08.041196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.041012 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 00:11:08.062253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.062206 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" podStartSLOduration=2.183045105 podStartE2EDuration="5.062190992s" totalImagesPullingTime="2.879145887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:08.060500273 +0000 UTC m=+358.206194730" watchObservedRunningTime="2026-11-03 00:11:08.062190992 +0000 UTC m=+358.207885447" Nov 03 00:11:08.245530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.245500 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:08.275521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.275476 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t\" (UniqueName: \"kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t\") pod \"99b6d6fe-c32c-4445-b7ec-b35924e3a46e\" (UID: \"99b6d6fe-c32c-4445-b7ec-b35924e3a46e\") " Nov 03 00:11:08.277886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.277858 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t" pod "99b6d6fe-c32c-4445-b7ec-b35924e3a46e" (UID: "99b6d6fe-c32c-4445-b7ec-b35924e3a46e"). InnerVolumeSpecName "kube-api-access-dh45t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:11:08.376529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.376496 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh45t\" (UniqueName: \"kubernetes.io/projected/99b6d6fe-c32c-4445-b7ec-b35924e3a46e-kube-api-access-dh45t\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:11:08.956950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.956910 5144 generic.go:356] "Generic (PLEG): container finished" podID="99b6d6fe-c32c-4445-b7ec-b35924e3a46e" containerID="4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9" exitCode=0 Nov 03 00:11:08.957136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.956957 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" Nov 03 00:11:08.957136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.956988 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" event={"ID":"99b6d6fe-c32c-4445-b7ec-b35924e3a46e","Type":"ContainerDied","Data":"4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9"} Nov 03 00:11:08.957136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.957037 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-wzv6h" event={"ID":"99b6d6fe-c32c-4445-b7ec-b35924e3a46e","Type":"ContainerDied","Data":"bb16b21099c1b7dabfa115177eb7e305796798729d3f3043041c4acb9b89e2f3"} Nov 03 00:11:08.957136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.957056 5144 scope.go:122] "RemoveContainer" containerID="4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9" Nov 03 00:11:08.966615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.966595 5144 scope.go:122] "RemoveContainer" containerID="4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9" Nov 03 00:11:08.966942 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:11:08.966922 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9\": container with ID starting with 4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9 not found: ID does not exist" containerID="4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9" Nov 03 00:11:08.967012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:08.966952 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9"} err="failed to get container status \"4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9\": rpc error: code = NotFound desc = could not find container \"4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9\": container with ID starting with 4302a67d3b5c76671c9d2c14b472fe3eb6ebff0f0360bbb3deb38713e66b21f9 not found: ID does not exist" Nov 03 00:11:09.040932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:09.040897 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body= Nov 03 00:11:09.041118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:09.040956 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" Nov 03 00:11:09.141164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:09.141126 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzv6h"] Nov 03 00:11:09.142967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:09.142943 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-wzv6h"] Nov 03 00:11:09.994548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:09.994511 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99b6d6fe-c32c-4445-b7ec-b35924e3a46e" path="/var/lib/kubelet/pods/99b6d6fe-c32c-4445-b7ec-b35924e3a46e/volumes" Nov 03 00:11:10.044940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:10.044908 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:11:10.045303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:10.045280 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:11:10.045969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:11:10.045954 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:11:12.155541 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:11:12.155505 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:11:42.155410 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:11:42.155369 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:12:12.155208 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:12:12.155169 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:12:22.132965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.132928 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-d7ngr"] Nov 03 00:12:22.133397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.133342 5144 cpu_manager.go:414] "containerMap: removing container" podUID="99b6d6fe-c32c-4445-b7ec-b35924e3a46e" containerName="authorino" Nov 03 00:12:22.133397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.133354 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99b6d6fe-c32c-4445-b7ec-b35924e3a46e" containerName="authorino" Nov 03 00:12:22.133474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.133446 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="99b6d6fe-c32c-4445-b7ec-b35924e3a46e" containerName="authorino" Nov 03 00:12:22.136588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.136560 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.189258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.189219 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Nov 03 00:12:22.189258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.189229 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Nov 03 00:12:22.238006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.237970 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-d7ngr\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.238180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.238011 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkfc\" (UniqueName: \"kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc\") pod \"ai-gateway-controller-5d8dddb676-d7ngr\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.238896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.238854 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-hvngs\"" Nov 03 00:12:22.240841 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.240818 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-d7ngr"] Nov 03 00:12:22.338885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.338839 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-d7ngr\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.338885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.338889 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkfc\" (UniqueName: \"kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc\") pod \"ai-gateway-controller-5d8dddb676-d7ngr\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.341501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.341480 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-d7ngr\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.395542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.395460 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkfc\" (UniqueName: \"kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc\") pod \"ai-gateway-controller-5d8dddb676-d7ngr\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.449931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.449891 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:22.827234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:22.827209 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-d7ngr"] Nov 03 00:12:22.829221 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:12:22.829189 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod765d787b_9097_43ba_a884_f1d4ac480019.slice/crio-216bff6c8612ae530cae980808766bbef2311dc54403ea459da1a99a3dd308cb WatchSource:0}: Error finding container 216bff6c8612ae530cae980808766bbef2311dc54403ea459da1a99a3dd308cb: Status 404 returned error can't find the container with id 216bff6c8612ae530cae980808766bbef2311dc54403ea459da1a99a3dd308cb Nov 03 00:12:23.320824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:23.320785 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" event={"ID":"765d787b-9097-43ba-a884-f1d4ac480019","Type":"ContainerStarted","Data":"216bff6c8612ae530cae980808766bbef2311dc54403ea459da1a99a3dd308cb"} Nov 03 00:12:26.341376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:26.341333 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" event={"ID":"765d787b-9097-43ba-a884-f1d4ac480019","Type":"ContainerStarted","Data":"f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8"} Nov 03 00:12:26.341876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:26.341426 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:26.459452 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:26.459396 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" podStartSLOduration=1.570139953 podStartE2EDuration="4.459382661s" totalImagesPullingTime="2.889242708s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:26.458006319 +0000 UTC m=+436.603700786" watchObservedRunningTime="2026-11-03 00:12:26.459382661 +0000 UTC m=+436.605077118" Nov 03 00:12:30.049829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.049795 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn"] Nov 03 00:12:30.053472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.053453 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.110362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.110318 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn"] Nov 03 00:12:30.156877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.156846 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Nov 03 00:12:30.157022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.156881 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-n6gcp\"" Nov 03 00:12:30.207866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.207830 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vh6mt"] Nov 03 00:12:30.211542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.211515 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.222037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.222004 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vpp5\" (UniqueName: \"kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5\") pod \"payload-pre-processing-7b98bb678c-w6pcn\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.222184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.222168 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w6pcn\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.292088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.292054 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vh6mt"] Nov 03 00:12:30.323569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.323486 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vpp5\" (UniqueName: \"kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5\") pod \"payload-pre-processing-7b98bb678c-w6pcn\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.323569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.323553 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vh6mt\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.323807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.323657 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w6pcn\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.323807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.323692 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7c7m\" (UniqueName: \"kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m\") pod \"payload-processing-55fbbbf9dc-vh6mt\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.324369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.324349 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-w6pcn\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.381408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.381371 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vpp5\" (UniqueName: \"kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5\") pod \"payload-pre-processing-7b98bb678c-w6pcn\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.424608 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.424576 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vh6mt\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.424806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.424699 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7c7m\" (UniqueName: \"kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m\") pod \"payload-processing-55fbbbf9dc-vh6mt\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.425281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.425259 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-vh6mt\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.482665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.482616 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7c7m\" (UniqueName: \"kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m\") pod \"payload-processing-55fbbbf9dc-vh6mt\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.532986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.532955 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:30.665921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.665885 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:30.902308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:30.902281 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vh6mt"] Nov 03 00:12:30.904284 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:12:30.904255 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod308a18b8_bcee_45dc_b1e5_4b10fcabc3c1.slice/crio-cef42142fef73755da52c6f1eaade348f3ab767001993fc702e7e59f1c1ac861 WatchSource:0}: Error finding container cef42142fef73755da52c6f1eaade348f3ab767001993fc702e7e59f1c1ac861: Status 404 returned error can't find the container with id cef42142fef73755da52c6f1eaade348f3ab767001993fc702e7e59f1c1ac861 Nov 03 00:12:31.038468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:31.038440 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn"] Nov 03 00:12:31.040413 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:12:31.040385 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83ca4574_d201_4912_9a0a_ddaadb5d598e.slice/crio-ae2b637319d2df00208428c88206cbd4f4d9beef1d9bfc35199269cc50ff81a3 WatchSource:0}: Error finding container ae2b637319d2df00208428c88206cbd4f4d9beef1d9bfc35199269cc50ff81a3: Status 404 returned error can't find the container with id ae2b637319d2df00208428c88206cbd4f4d9beef1d9bfc35199269cc50ff81a3 Nov 03 00:12:31.367556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:31.367517 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" event={"ID":"83ca4574-d201-4912-9a0a-ddaadb5d598e","Type":"ContainerStarted","Data":"ae2b637319d2df00208428c88206cbd4f4d9beef1d9bfc35199269cc50ff81a3"} Nov 03 00:12:31.368711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:31.368688 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" event={"ID":"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1","Type":"ContainerStarted","Data":"cef42142fef73755da52c6f1eaade348f3ab767001993fc702e7e59f1c1ac861"} Nov 03 00:12:34.389338 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:34.389292 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" event={"ID":"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1","Type":"ContainerStarted","Data":"82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3"} Nov 03 00:12:34.389850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:34.389537 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:34.391109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:34.391080 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" event={"ID":"83ca4574-d201-4912-9a0a-ddaadb5d598e","Type":"ContainerStarted","Data":"65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e"} Nov 03 00:12:34.391243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:34.391177 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:34.512502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:34.512426 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" podStartSLOduration=1.5270474520000001 podStartE2EDuration="4.5124075s" totalImagesPullingTime="2.985360048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:34.510864058 +0000 UTC m=+444.656558517" watchObservedRunningTime="2026-11-03 00:12:34.5124075 +0000 UTC m=+444.658101962" Nov 03 00:12:34.625018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:34.624957 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" podStartSLOduration=1.807829725 podStartE2EDuration="4.624940067s" totalImagesPullingTime="2.817110342s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:34.623577821 +0000 UTC m=+444.769272301" watchObservedRunningTime="2026-11-03 00:12:34.624940067 +0000 UTC m=+444.770634532" Nov 03 00:12:37.349181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.349150 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:37.721489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.721414 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-67cc775cd9-n49hj"] Nov 03 00:12:37.725100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.725079 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:37.781690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.781657 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67cc775cd9-n49hj"] Nov 03 00:12:37.897190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.897149 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfkw5\" (UniqueName: \"kubernetes.io/projected/af51a4fd-3ae1-46ac-b602-208c8a54c058-kube-api-access-lfkw5\") pod \"ai-gateway-controller-67cc775cd9-n49hj\" (UID: \"af51a4fd-3ae1-46ac-b602-208c8a54c058\") " pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:37.897375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.897202 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af51a4fd-3ae1-46ac-b602-208c8a54c058-metrics-certs\") pod \"ai-gateway-controller-67cc775cd9-n49hj\" (UID: \"af51a4fd-3ae1-46ac-b602-208c8a54c058\") " pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:37.998571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.998493 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfkw5\" (UniqueName: \"kubernetes.io/projected/af51a4fd-3ae1-46ac-b602-208c8a54c058-kube-api-access-lfkw5\") pod \"ai-gateway-controller-67cc775cd9-n49hj\" (UID: \"af51a4fd-3ae1-46ac-b602-208c8a54c058\") " pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:37.998571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:37.998534 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af51a4fd-3ae1-46ac-b602-208c8a54c058-metrics-certs\") pod \"ai-gateway-controller-67cc775cd9-n49hj\" (UID: \"af51a4fd-3ae1-46ac-b602-208c8a54c058\") " pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:38.001134 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:38.001110 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/af51a4fd-3ae1-46ac-b602-208c8a54c058-metrics-certs\") pod \"ai-gateway-controller-67cc775cd9-n49hj\" (UID: \"af51a4fd-3ae1-46ac-b602-208c8a54c058\") " pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:38.055542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:38.055506 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfkw5\" (UniqueName: \"kubernetes.io/projected/af51a4fd-3ae1-46ac-b602-208c8a54c058-kube-api-access-lfkw5\") pod \"ai-gateway-controller-67cc775cd9-n49hj\" (UID: \"af51a4fd-3ae1-46ac-b602-208c8a54c058\") " pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:38.338251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:38.338210 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:38.708747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:38.708723 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-67cc775cd9-n49hj"] Nov 03 00:12:38.710439 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:12:38.710400 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf51a4fd_3ae1_46ac_b602_208c8a54c058.slice/crio-0d4e5b3913dc5f00257fd2d9c29da953494f9608f5e52506280663f48ab8b97c WatchSource:0}: Error finding container 0d4e5b3913dc5f00257fd2d9c29da953494f9608f5e52506280663f48ab8b97c: Status 404 returned error can't find the container with id 0d4e5b3913dc5f00257fd2d9c29da953494f9608f5e52506280663f48ab8b97c Nov 03 00:12:39.419763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:39.419727 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" event={"ID":"af51a4fd-3ae1-46ac-b602-208c8a54c058","Type":"ContainerStarted","Data":"0d4e5b3913dc5f00257fd2d9c29da953494f9608f5e52506280663f48ab8b97c"} Nov 03 00:12:40.426790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:40.426749 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" event={"ID":"af51a4fd-3ae1-46ac-b602-208c8a54c058","Type":"ContainerStarted","Data":"8a55e0fc789cca6898089325c49d75a5f93c180b10a37615c2071eb0bd89e167"} Nov 03 00:12:40.427294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:40.426940 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:40.540931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:40.540866 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" podStartSLOduration=2.740599951 podStartE2EDuration="3.540851774s" totalImagesPullingTime="800.251823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:40.540416903 +0000 UTC m=+450.686111360" watchObservedRunningTime="2026-11-03 00:12:40.540851774 +0000 UTC m=+450.686546230" Nov 03 00:12:42.155390 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:12:42.155352 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:12:45.399737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:45.399708 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 00:12:45.400210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:45.399779 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 00:12:51.435902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.435871 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-67cc775cd9-n49hj" Nov 03 00:12:51.569907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.569873 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-d7ngr"] Nov 03 00:12:51.570105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.570078 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" podUID="765d787b-9097-43ba-a884-f1d4ac480019" containerName="manager" containerID="cri-o://f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8" gracePeriod=30 Nov 03 00:12:51.823704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.823681 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:51.924368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.924334 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs\" (UniqueName: \"kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs\") pod \"765d787b-9097-43ba-a884-f1d4ac480019\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " Nov 03 00:12:51.924529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.924385 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc\" (UniqueName: \"kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc\") pod \"765d787b-9097-43ba-a884-f1d4ac480019\" (UID: \"765d787b-9097-43ba-a884-f1d4ac480019\") " Nov 03 00:12:51.926711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.926679 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs" pod "765d787b-9097-43ba-a884-f1d4ac480019" (UID: "765d787b-9097-43ba-a884-f1d4ac480019"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:12:51.926839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:51.926744 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc" pod "765d787b-9097-43ba-a884-f1d4ac480019" (UID: "765d787b-9097-43ba-a884-f1d4ac480019"). InnerVolumeSpecName "kube-api-access-djkfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:12:52.025572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.025543 5144 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/765d787b-9097-43ba-a884-f1d4ac480019-metrics-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:52.025572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.025571 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djkfc\" (UniqueName: \"kubernetes.io/projected/765d787b-9097-43ba-a884-f1d4ac480019-kube-api-access-djkfc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:12:52.492083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.492047 5144 generic.go:356] "Generic (PLEG): container finished" podID="765d787b-9097-43ba-a884-f1d4ac480019" containerID="f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8" exitCode=0 Nov 03 00:12:52.492515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.492096 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" Nov 03 00:12:52.492515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.492131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" event={"ID":"765d787b-9097-43ba-a884-f1d4ac480019","Type":"ContainerDied","Data":"f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8"} Nov 03 00:12:52.492515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.492184 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-d7ngr" event={"ID":"765d787b-9097-43ba-a884-f1d4ac480019","Type":"ContainerDied","Data":"216bff6c8612ae530cae980808766bbef2311dc54403ea459da1a99a3dd308cb"} Nov 03 00:12:52.492515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.492206 5144 scope.go:122] "RemoveContainer" containerID="f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8" Nov 03 00:12:52.501878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.501857 5144 scope.go:122] "RemoveContainer" containerID="f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8" Nov 03 00:12:52.502176 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:12:52.502155 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8\": container with ID starting with f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8 not found: ID does not exist" containerID="f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8" Nov 03 00:12:52.502279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.502183 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8"} err="failed to get container status \"f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8\": rpc error: code = NotFound desc = could not find container \"f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8\": container with ID starting with f73adf7420c8fcce11b5ea364f9ef0fdda186d3b54c3ee45d3991a4a0382adc8 not found: ID does not exist" Nov 03 00:12:52.654990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.654955 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-d7ngr"] Nov 03 00:12:52.704553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:52.704516 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-d7ngr"] Nov 03 00:12:53.995958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:12:53.995921 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="765d787b-9097-43ba-a884-f1d4ac480019" path="/var/lib/kubelet/pods/765d787b-9097-43ba-a884-f1d4ac480019/volumes" Nov 03 00:13:11.937964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.937931 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws"] Nov 03 00:13:11.938379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.938366 5144 cpu_manager.go:414] "containerMap: removing container" podUID="765d787b-9097-43ba-a884-f1d4ac480019" containerName="manager" Nov 03 00:13:11.938432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.938380 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="765d787b-9097-43ba-a884-f1d4ac480019" containerName="manager" Nov 03 00:13:11.938475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.938465 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="765d787b-9097-43ba-a884-f1d4ac480019" containerName="manager" Nov 03 00:13:11.943703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.943682 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:11.995193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.995163 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws"] Nov 03 00:13:11.995947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:11.995925 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 00:13:12.039533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.039499 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ggwn7\"" Nov 03 00:13:12.039533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.039505 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 00:13:12.039753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.039736 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Nov 03 00:13:12.114341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.114309 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjv6\" (UniqueName: \"kubernetes.io/projected/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-kube-api-access-cvjv6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.114548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.114352 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.114548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.114374 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.114548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.114388 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.114770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.114601 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.114770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.114679 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.155281 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:13:12.155244 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:13:12.216023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.215926 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.215967 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216088 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjv6\" (UniqueName: \"kubernetes.io/projected/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-kube-api-access-cvjv6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216120 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216147 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216167 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216482 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216397 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216481 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.216600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.216582 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.218791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.218766 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.218791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.218782 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.273237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.273209 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjv6\" (UniqueName: \"kubernetes.io/projected/f1b7db33-4fcc-421a-b0f4-5a14c5da0312-kube-api-access-cvjv6\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-kqqws\" (UID: \"f1b7db33-4fcc-421a-b0f4-5a14c5da0312\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.555490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.555452 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:12.928266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:12.928241 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws"] Nov 03 00:13:12.930664 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:13:12.930621 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b7db33_4fcc_421a_b0f4_5a14c5da0312.slice/crio-69d98eec65f8e2504eb935c48fd499e28a04124c119632f5c4dda2409f307bed WatchSource:0}: Error finding container 69d98eec65f8e2504eb935c48fd499e28a04124c119632f5c4dda2409f307bed: Status 404 returned error can't find the container with id 69d98eec65f8e2504eb935c48fd499e28a04124c119632f5c4dda2409f307bed Nov 03 00:13:13.621922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:13.621884 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" event={"ID":"f1b7db33-4fcc-421a-b0f4-5a14c5da0312","Type":"ContainerStarted","Data":"69d98eec65f8e2504eb935c48fd499e28a04124c119632f5c4dda2409f307bed"} Nov 03 00:13:15.564171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.564135 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj"] Nov 03 00:13:15.567047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.567022 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.637983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.637946 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" event={"ID":"f1b7db33-4fcc-421a-b0f4-5a14c5da0312","Type":"ContainerStarted","Data":"294c4a08e5d5381a2f615ec0df46dc8870861e0e5151447bb745239b4da230c8"} Nov 03 00:13:15.670645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.670608 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 00:13:15.672932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.672913 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj"] Nov 03 00:13:15.749546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.749500 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.749766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.749656 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.749766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.749691 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.749766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.749717 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.749766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.749738 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq6w4\" (UniqueName: \"kubernetes.io/projected/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-kube-api-access-qq6w4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.749992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.749863 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.796070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.796006 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" podStartSLOduration=3.168824735 podStartE2EDuration="4.795992954s" totalImagesPullingTime="1.627168219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:15.795180776 +0000 UTC m=+485.940875256" watchObservedRunningTime="2026-11-03 00:13:15.795992954 +0000 UTC m=+485.941687411" Nov 03 00:13:15.850713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.850614 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.850713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.850681 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.850901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.850713 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.850901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.850738 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq6w4\" (UniqueName: \"kubernetes.io/projected/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-kube-api-access-qq6w4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.850901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.850790 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.850901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.850871 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.851107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.851046 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.851107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.851068 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.851216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.851143 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.853338 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.853306 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.853478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.853461 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:15.908263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:15.908226 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq6w4\" (UniqueName: \"kubernetes.io/projected/c8dd533d-25a0-4034-bfbe-c9f9808a5cce-kube-api-access-qq6w4\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-k4hrj\" (UID: \"c8dd533d-25a0-4034-bfbe-c9f9808a5cce\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:16.178850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:16.178767 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:16.552692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:16.552535 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj"] Nov 03 00:13:16.555606 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:16.555573 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:16.555825 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:13:16.555797 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8dd533d_25a0_4034_bfbe_c9f9808a5cce.slice/crio-97976636ff79fb638e6d80201438d852eb4124192838db96da8ea896452685c3 WatchSource:0}: Error finding container 97976636ff79fb638e6d80201438d852eb4124192838db96da8ea896452685c3: Status 404 returned error can't find the container with id 97976636ff79fb638e6d80201438d852eb4124192838db96da8ea896452685c3 Nov 03 00:13:16.646611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:16.646558 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" event={"ID":"c8dd533d-25a0-4034-bfbe-c9f9808a5cce","Type":"ContainerStarted","Data":"97976636ff79fb638e6d80201438d852eb4124192838db96da8ea896452685c3"} Nov 03 00:13:17.654709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:17.654578 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" event={"ID":"c8dd533d-25a0-4034-bfbe-c9f9808a5cce","Type":"ContainerStarted","Data":"e5e72157f1034753877bddd2f01b038b392237aef490cac9a1c86e8f53be1941"} Nov 03 00:13:17.769172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:17.769116 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" podStartSLOduration=2.041202637 podStartE2EDuration="2.769101299s" totalImagesPullingTime="727.898662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:17.768198396 +0000 UTC m=+487.913892853" watchObservedRunningTime="2026-11-03 00:13:17.769101299 +0000 UTC m=+487.914795791" Nov 03 00:13:18.179555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:18.179516 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:22.556691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.556597 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:22.568940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.568916 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:22.693277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.693249 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-kqqws" Nov 03 00:13:22.789241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.789210 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tsv68"] Nov 03 00:13:22.792798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.792778 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:22.849184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.846688 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 00:13:22.853351 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.853322 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tsv68"] Nov 03 00:13:22.919734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.919701 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:22.919922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.919751 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-home\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:22.919922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.919873 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:22.920037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.919920 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncgk2\" (UniqueName: \"kubernetes.io/projected/83150290-61b8-4483-8c83-ecef4a633e4b-kube-api-access-ncgk2\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:22.920075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.920031 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:22.920075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:22.920066 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83150290-61b8-4483-8c83-ecef4a633e4b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.021432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.021376 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83150290-61b8-4483-8c83-ecef4a633e4b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.021612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.021541 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.021612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.021575 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-home\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.021758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.021623 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.021758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.021680 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncgk2\" (UniqueName: \"kubernetes.io/projected/83150290-61b8-4483-8c83-ecef4a633e4b-kube-api-access-ncgk2\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.021758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.021747 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.022121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.022087 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.022275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.022252 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.022467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.022448 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-home\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.024670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.024614 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83150290-61b8-4483-8c83-ecef4a633e4b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.024762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.024706 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83150290-61b8-4483-8c83-ecef4a633e4b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.080342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.080303 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncgk2\" (UniqueName: \"kubernetes.io/projected/83150290-61b8-4483-8c83-ecef4a633e4b-kube-api-access-ncgk2\") pod \"echo-simulator-kserve-64b6f8f96c-tsv68\" (UID: \"83150290-61b8-4483-8c83-ecef4a633e4b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.106023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.105944 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:23.483197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.483170 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-tsv68"] Nov 03 00:13:23.485547 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:13:23.485521 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83150290_61b8_4483_8c83_ecef4a633e4b.slice/crio-70a6bf5b9e48ca514b064fbaf599e0dd61b10dbb4329fe1744cbfa72e3672750 WatchSource:0}: Error finding container 70a6bf5b9e48ca514b064fbaf599e0dd61b10dbb4329fe1744cbfa72e3672750: Status 404 returned error can't find the container with id 70a6bf5b9e48ca514b064fbaf599e0dd61b10dbb4329fe1744cbfa72e3672750 Nov 03 00:13:23.688425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:23.688388 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" event={"ID":"83150290-61b8-4483-8c83-ecef4a633e4b","Type":"ContainerStarted","Data":"70a6bf5b9e48ca514b064fbaf599e0dd61b10dbb4329fe1744cbfa72e3672750"} Nov 03 00:13:26.180072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.180039 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:26.194093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.194067 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:26.423233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.423191 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc"] Nov 03 00:13:26.427310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.427278 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.456483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.456440 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.456722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.456503 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0129e1-2867-42f9-862b-96bf78fe011d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.456722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.456582 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.456722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.456611 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.456722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.456692 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.456953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.456752 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54gx\" (UniqueName: \"kubernetes.io/projected/5b0129e1-2867-42f9-862b-96bf78fe011d-kube-api-access-c54gx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.480151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.480112 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 00:13:26.482621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.482590 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc"] Nov 03 00:13:26.558440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.558167 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.558673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.558474 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.558673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.558530 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.558673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.558595 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54gx\" (UniqueName: \"kubernetes.io/projected/5b0129e1-2867-42f9-862b-96bf78fe011d-kube-api-access-c54gx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.558855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.558690 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.558855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.558724 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0129e1-2867-42f9-862b-96bf78fe011d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.559079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.559057 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.559576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.559537 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.560026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.559833 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.560824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.560798 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b0129e1-2867-42f9-862b-96bf78fe011d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.562123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.562101 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b0129e1-2867-42f9-862b-96bf78fe011d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.615704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.615664 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54gx\" (UniqueName: \"kubernetes.io/projected/5b0129e1-2867-42f9-862b-96bf78fe011d-kube-api-access-c54gx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc\" (UID: \"5b0129e1-2867-42f9-862b-96bf78fe011d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.707935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.707835 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" event={"ID":"83150290-61b8-4483-8c83-ecef4a633e4b","Type":"ContainerStarted","Data":"b95299bf852acaf451689ca2d0e7efc3c6fd1d64f8f3f776480567ec6b8e779e"} Nov 03 00:13:26.719767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.719739 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-k4hrj" Nov 03 00:13:26.744062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.744019 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:26.938263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:26.938204 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" podStartSLOduration=1.8474870239999999 podStartE2EDuration="4.938188396s" totalImagesPullingTime="3.090701372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:26.822924883 +0000 UTC m=+496.968619349" watchObservedRunningTime="2026-11-03 00:13:26.938188396 +0000 UTC m=+497.083882907" Nov 03 00:13:27.106291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:27.106255 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:27.124255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:27.124231 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc"] Nov 03 00:13:27.125945 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:13:27.125916 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0129e1_2867_42f9_862b_96bf78fe011d.slice/crio-70221e4fefada16cb88db76542885851a9c5ef51825ff24596989d9d05754d93 WatchSource:0}: Error finding container 70221e4fefada16cb88db76542885851a9c5ef51825ff24596989d9d05754d93: Status 404 returned error can't find the container with id 70221e4fefada16cb88db76542885851a9c5ef51825ff24596989d9d05754d93 Nov 03 00:13:27.715386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:27.715348 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" event={"ID":"5b0129e1-2867-42f9-862b-96bf78fe011d","Type":"ContainerStarted","Data":"70221e4fefada16cb88db76542885851a9c5ef51825ff24596989d9d05754d93"} Nov 03 00:13:28.723451 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:28.723417 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" event={"ID":"5b0129e1-2867-42f9-862b-96bf78fe011d","Type":"ContainerStarted","Data":"85ccbd6d883e032a6e38fad639c3bfc487cc178fdb836ed4026467e4e4993474"} Nov 03 00:13:28.744573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:28.744541 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:28.838993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:28.838935 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" podStartSLOduration=2.125020294 podStartE2EDuration="2.838919394s" totalImagesPullingTime="713.8991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:28.836289584 +0000 UTC m=+498.981984060" watchObservedRunningTime="2026-11-03 00:13:28.838919394 +0000 UTC m=+498.984613868" Nov 03 00:13:33.106914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:33.106873 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:33.119981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:33.119947 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:33.765896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:33.765869 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-tsv68" Nov 03 00:13:36.745176 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:36.745134 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:36.758690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:36.758652 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:36.782275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:36.782244 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc" Nov 03 00:13:42.154790 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:13:42.154746 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:13:49.387024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.386988 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k"] Nov 03 00:13:49.393017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.392998 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.473609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.473577 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k"] Nov 03 00:13:49.481488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481460 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.481656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481522 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gzbg\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.481707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481677 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.481741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481726 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.481819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481802 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.481853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481826 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.481893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481854 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.482000 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.481978 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.482062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.482012 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583033 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gzbg\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583079 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583100 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583130 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583149 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583175 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583208 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583231 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583297 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583683 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583731 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583784 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.583840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.583812 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.584149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.584127 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.585696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.585677 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.585899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.585880 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.645747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.645667 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.645912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.645752 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gzbg\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-rln6k\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:49.705931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:49.705893 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:50.181667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.181622 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k"] Nov 03 00:13:50.183574 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:13:50.183542 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbde53d5_8b8d_49a2_b9eb_b8a852a60e88.slice/crio-bc61695167a6a56a9340ded55180eeebfe49d92c40b17bad96c5f618a15ce2db WatchSource:0}: Error finding container bc61695167a6a56a9340ded55180eeebfe49d92c40b17bad96c5f618a15ce2db: Status 404 returned error can't find the container with id bc61695167a6a56a9340ded55180eeebfe49d92c40b17bad96c5f618a15ce2db Nov 03 00:13:50.185839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.185806 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:13:50.185900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.185879 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:13:50.185939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.185907 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:13:50.850830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.850790 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" event={"ID":"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88","Type":"ContainerStarted","Data":"44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80"} Nov 03 00:13:50.851339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.850845 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" event={"ID":"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88","Type":"ContainerStarted","Data":"bc61695167a6a56a9340ded55180eeebfe49d92c40b17bad96c5f618a15ce2db"} Nov 03 00:13:50.967575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:50.967513 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podStartSLOduration=1.9674974710000002 podStartE2EDuration="1.967497471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:50.966217372 +0000 UTC m=+521.111911826" watchObservedRunningTime="2026-11-03 00:13:50.967497471 +0000 UTC m=+521.113191929" Nov 03 00:13:51.706243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:51.706205 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:51.707886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:51.707862 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:13:51.708012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:51.707908 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 00:13:52.706697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:52.706658 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:13:52.707138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:52.706718 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 00:13:53.706841 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:53.706805 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:13:53.707258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:53.706860 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 00:13:54.706700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:54.706666 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:13:54.706885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:54.706725 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 00:13:55.707075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:55.707038 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:13:55.707497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:55.707093 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 00:13:56.707219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:56.707175 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Nov 03 00:13:56.707689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:56.707247 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Nov 03 00:13:57.545102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.545070 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-64cfdbc956-4ltf2"] Nov 03 00:13:57.553085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.553067 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.603923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.603887 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64cfdbc956-4ltf2"] Nov 03 00:13:57.654187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.654155 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 00:13:57.654187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.654163 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7ns8x\"" Nov 03 00:13:57.687448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.687396 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2768863-e2e0-46bd-9369-58c6a8a140d6-tls-cert\") pod \"authorino-64cfdbc956-4ltf2\" (UID: \"a2768863-e2e0-46bd-9369-58c6a8a140d6\") " pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.687684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.687603 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4kk5\" (UniqueName: \"kubernetes.io/projected/a2768863-e2e0-46bd-9369-58c6a8a140d6-kube-api-access-q4kk5\") pod \"authorino-64cfdbc956-4ltf2\" (UID: \"a2768863-e2e0-46bd-9369-58c6a8a140d6\") " pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.788858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.788819 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4kk5\" (UniqueName: \"kubernetes.io/projected/a2768863-e2e0-46bd-9369-58c6a8a140d6-kube-api-access-q4kk5\") pod \"authorino-64cfdbc956-4ltf2\" (UID: \"a2768863-e2e0-46bd-9369-58c6a8a140d6\") " pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.789327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.788904 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2768863-e2e0-46bd-9369-58c6a8a140d6-tls-cert\") pod \"authorino-64cfdbc956-4ltf2\" (UID: \"a2768863-e2e0-46bd-9369-58c6a8a140d6\") " pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.791562 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.791533 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a2768863-e2e0-46bd-9369-58c6a8a140d6-tls-cert\") pod \"authorino-64cfdbc956-4ltf2\" (UID: \"a2768863-e2e0-46bd-9369-58c6a8a140d6\") " pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.845602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.845518 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4kk5\" (UniqueName: \"kubernetes.io/projected/a2768863-e2e0-46bd-9369-58c6a8a140d6-kube-api-access-q4kk5\") pod \"authorino-64cfdbc956-4ltf2\" (UID: \"a2768863-e2e0-46bd-9369-58c6a8a140d6\") " pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:57.865406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:57.865373 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-64cfdbc956-4ltf2" Nov 03 00:13:58.250456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:58.250362 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-64cfdbc956-4ltf2"] Nov 03 00:13:58.254113 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:13:58.254082 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda2768863_e2e0_46bd_9369_58c6a8a140d6.slice/crio-3f73507be9d5be9b8289a89d6a082ccacbcfe4d74d546fac883eb624b65b20f4 WatchSource:0}: Error finding container 3f73507be9d5be9b8289a89d6a082ccacbcfe4d74d546fac883eb624b65b20f4: Status 404 returned error can't find the container with id 3f73507be9d5be9b8289a89d6a082ccacbcfe4d74d546fac883eb624b65b20f4 Nov 03 00:13:58.419961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:58.419928 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:58.420330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:58.420305 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:58.421261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:58.421240 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:13:58.895674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:58.895625 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64cfdbc956-4ltf2" event={"ID":"a2768863-e2e0-46bd-9369-58c6a8a140d6","Type":"ContainerStarted","Data":"3f73507be9d5be9b8289a89d6a082ccacbcfe4d74d546fac883eb624b65b20f4"} Nov 03 00:13:59.902388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:13:59.902339 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-64cfdbc956-4ltf2" event={"ID":"a2768863-e2e0-46bd-9369-58c6a8a140d6","Type":"ContainerStarted","Data":"b1c03f8915994f56db0d49ce2774e83ff5839c6e792ce8bfd9cca7ec97d8cabf"} Nov 03 00:14:00.016959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:00.016802 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-64cfdbc956-4ltf2" podStartSLOduration=1.515716464 podStartE2EDuration="3.016783653s" totalImagesPullingTime="1.501067189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:00.014147322 +0000 UTC m=+530.159841794" watchObservedRunningTime="2026-11-03 00:14:00.016783653 +0000 UTC m=+530.162478110" Nov 03 00:14:12.154719 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:14:12.154677 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:14:36.178825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.178788 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws"] Nov 03 00:14:36.183019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.182996 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.234622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.234589 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2ccce380-openshift-default-dockercfg-ts9jv\"" Nov 03 00:14:36.280426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280357 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280499 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280535 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280581 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktv4b\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280680 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280760 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280800 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.280969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280884 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.281020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.280970 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.290148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.290109 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws"] Nov 03 00:14:36.382101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382062 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382120 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382149 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382166 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382189 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktv4b\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382216 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382236 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382337 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382429 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382675 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382697 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382894 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.382934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.382927 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.383280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.383254 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.384829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.384794 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.384998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.384978 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.442272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.442174 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktv4b\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.443867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.442591 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token\") pod \"e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.482678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.482622 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24"] Nov 03 00:14:36.487904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.487879 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.495794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.495771 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:36.541673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.541541 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-01450ec4-openshift-default-dockercfg-j2zzf\"" Nov 03 00:14:36.545836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.545804 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24"] Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584061 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584135 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584235 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584272 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584312 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584345 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584393 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584433 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5f6z\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.584680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.584469 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685433 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685479 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685533 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685554 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685588 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685620 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.685702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685678 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685713 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5f6z\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685892 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.685955 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.686529 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.686689 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.686855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.686833 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.688237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.688217 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.688456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.688439 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.743031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.742941 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.743178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.743079 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5f6z\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z\") pod \"e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.802678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.802608 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:36.960209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.960185 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws"] Nov 03 00:14:36.961522 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:14:36.961492 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a55c316_a586_4377_99eb_164c028773cf.slice/crio-2bada98383dd68f6510c9bb8b507c53d1ff63bcbd616c78ac17fd3db927f1558 WatchSource:0}: Error finding container 2bada98383dd68f6510c9bb8b507c53d1ff63bcbd616c78ac17fd3db927f1558: Status 404 returned error can't find the container with id 2bada98383dd68f6510c9bb8b507c53d1ff63bcbd616c78ac17fd3db927f1558 Nov 03 00:14:36.963868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.963838 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:36.963954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.963902 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:36.963954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:36.963934 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:37.111928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.111891 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" event={"ID":"2a55c316-a586-4377-99eb-164c028773cf","Type":"ContainerStarted","Data":"04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2"} Nov 03 00:14:37.112146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.111949 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" event={"ID":"2a55c316-a586-4377-99eb-164c028773cf","Type":"ContainerStarted","Data":"2bada98383dd68f6510c9bb8b507c53d1ff63bcbd616c78ac17fd3db927f1558"} Nov 03 00:14:37.112298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.112235 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd"] Nov 03 00:14:37.116188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.116165 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.170507 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.170462 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-c6f27688-gw-openshift-default-dockercfg-zfpsw\"" Nov 03 00:14:37.173416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.173381 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd"] Nov 03 00:14:37.186774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.186739 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24"] Nov 03 00:14:37.190252 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:14:37.190217 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72919027_3c7d_4b04_8fd1_5f92fcaf1b4f.slice/crio-c93facfd545d62a44cb5becf94816ad771df4622ae9130f718bd476787616c1d WatchSource:0}: Error finding container c93facfd545d62a44cb5becf94816ad771df4622ae9130f718bd476787616c1d: Status 404 returned error can't find the container with id c93facfd545d62a44cb5becf94816ad771df4622ae9130f718bd476787616c1d Nov 03 00:14:37.192472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.192429 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:37.192575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.192524 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:37.192658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.192577 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:37.237785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.237729 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podStartSLOduration=1.237711882 podStartE2EDuration="1.237711882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:37.235614097 +0000 UTC m=+567.381308580" watchObservedRunningTime="2026-11-03 00:14:37.237711882 +0000 UTC m=+567.383406337" Nov 03 00:14:37.295490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295450 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.295669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295496 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.295669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295580 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.295669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295651 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.295876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295760 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.295876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295845 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.295981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295957 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.296036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.295989 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.296307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.296271 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrps7\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.396958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.396915 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.396958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.396959 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.396999 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397148 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397228 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397284 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397350 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397380 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397438 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrps7\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397468 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.397904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397882 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.398009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.397882 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.398292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.398268 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.398497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.398477 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.399848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.399827 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.400098 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.400082 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.454890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.454853 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.455056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.454951 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrps7\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7\") pod \"e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:37.495960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.495921 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:37.733427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:37.733334 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:38.119659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.119604 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" event={"ID":"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f","Type":"ContainerStarted","Data":"a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790"} Nov 03 00:14:38.119859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.119696 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" event={"ID":"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f","Type":"ContainerStarted","Data":"c93facfd545d62a44cb5becf94816ad771df4622ae9130f718bd476787616c1d"} Nov 03 00:14:38.160756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.160728 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd"] Nov 03 00:14:38.161861 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:14:38.161834 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae357a1_8b3d_4210_b7f3_0edb0b961106.slice/crio-00debc95664f00f26d176f4073a2a859c1b6ba6c5ff66e8514cd55a493039807 WatchSource:0}: Error finding container 00debc95664f00f26d176f4073a2a859c1b6ba6c5ff66e8514cd55a493039807: Status 404 returned error can't find the container with id 00debc95664f00f26d176f4073a2a859c1b6ba6c5ff66e8514cd55a493039807 Nov 03 00:14:38.164199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.164160 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:38.164303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.164242 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:38.164303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.164286 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:38.234918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.234856 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podStartSLOduration=2.234835733 podStartE2EDuration="2.234835733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:38.233942072 +0000 UTC m=+568.379636545" watchObservedRunningTime="2026-11-03 00:14:38.234835733 +0000 UTC m=+568.380530192" Nov 03 00:14:38.502095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.502021 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:38.566854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.566822 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74"] Nov 03 00:14:38.570685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.570668 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611394 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611427 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611448 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611537 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611615 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611727 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611755 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgjsx\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611813 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.611943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.611920 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.627685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.627019 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-12537192-openshift-default-dockercfg-9rlvz\"" Nov 03 00:14:38.630326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.630298 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74"] Nov 03 00:14:38.712527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712496 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.712742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712557 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.712742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712578 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgjsx\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.712742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712612 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.712742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712732 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712763 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712786 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712815 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.712838 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.713037 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.713146 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.713178 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.713230 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.713672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.713579 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.715349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.715328 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.715543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.715528 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.769734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.769702 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.769886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.769794 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgjsx\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx\") pod \"e2e-ait-12537192-openshift-default-546d597f7-h6h74\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:38.809041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.807934 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:38.818817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.818233 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:38.884725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:38.884690 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:39.126511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.126398 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" event={"ID":"3ae357a1-8b3d-4210-b7f3-0edb0b961106","Type":"ContainerStarted","Data":"eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd"} Nov 03 00:14:39.126511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.126460 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" event={"ID":"3ae357a1-8b3d-4210-b7f3-0edb0b961106","Type":"ContainerStarted","Data":"00debc95664f00f26d176f4073a2a859c1b6ba6c5ff66e8514cd55a493039807"} Nov 03 00:14:39.127030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.126948 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:39.127030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.126994 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:39.127941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.127921 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:14:39.128042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.128017 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:14:39.244681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.244601 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podStartSLOduration=2.244585746 podStartE2EDuration="2.244585746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:39.242869038 +0000 UTC m=+569.388563495" watchObservedRunningTime="2026-11-03 00:14:39.244585746 +0000 UTC m=+569.390280193" Nov 03 00:14:39.312227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.312201 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74"] Nov 03 00:14:39.313100 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:14:39.313073 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6dc027a_bd51_43db_91a8_5cca7787a86b.slice/crio-97e79f2727540300fddc47e89cda6cae844f7d646e1bd36659db22b4a37c8285 WatchSource:0}: Error finding container 97e79f2727540300fddc47e89cda6cae844f7d646e1bd36659db22b4a37c8285: Status 404 returned error can't find the container with id 97e79f2727540300fddc47e89cda6cae844f7d646e1bd36659db22b4a37c8285 Nov 03 00:14:39.315382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.315352 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:39.315442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.315413 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:39.315479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.315450 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:14:39.733695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.733608 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:39.738891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:39.738865 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:40.133916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.133884 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" event={"ID":"c6dc027a-bd51-43db-91a8-5cca7787a86b","Type":"ContainerStarted","Data":"f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d"} Nov 03 00:14:40.134138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.133946 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" event={"ID":"c6dc027a-bd51-43db-91a8-5cca7787a86b","Type":"ContainerStarted","Data":"97e79f2727540300fddc47e89cda6cae844f7d646e1bd36659db22b4a37c8285"} Nov 03 00:14:40.134685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.134661 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:40.135724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.135704 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:14:40.256789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.256740 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podStartSLOduration=2.256722431 podStartE2EDuration="2.256722431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:40.253545793 +0000 UTC m=+570.399240251" watchObservedRunningTime="2026-11-03 00:14:40.256722431 +0000 UTC m=+570.402416885" Nov 03 00:14:40.885260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.885225 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:40.890271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:40.890247 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:41.139813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:41.139735 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:41.140626 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:14:41.140610 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:14:42.156791 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:14:42.156756 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:15:00.157912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.157877 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx"] Nov 03 00:15:00.162072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.162050 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.164293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.164270 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j"] Nov 03 00:15:00.168517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.168497 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr"] Nov 03 00:15:00.168641 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.168612 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.172084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.172069 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.229943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.229895 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:15:00.258179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.258148 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx"] Nov 03 00:15:00.266303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.266271 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j"] Nov 03 00:15:00.298690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.294089 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpkmk\" (UniqueName: \"kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk\") pod \"maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.298690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.294173 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlm7f\" (UniqueName: \"kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f\") pod \"maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.298690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.294296 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.298690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.294387 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.298690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.294430 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn827\" (UniqueName: \"kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827\") pod \"maas-api-key-cleanup-29894415-z4c6j\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.298690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.294475 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-z4c6j\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.344574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.344172 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr"] Nov 03 00:15:00.395913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.395884 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlm7f\" (UniqueName: \"kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f\") pod \"maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.396121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.395960 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.396121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396019 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.396121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396039 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn827\" (UniqueName: \"kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827\") pod \"maas-api-key-cleanup-29894415-z4c6j\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.396121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396063 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-z4c6j\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.396121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396084 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpkmk\" (UniqueName: \"kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk\") pod \"maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.396771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396751 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.396825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396761 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.396825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.396786 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894415-z4c6j\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.452623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.452551 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn827\" (UniqueName: \"kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827\") pod \"maas-api-key-cleanup-29894415-z4c6j\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.452989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.452965 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpkmk\" (UniqueName: \"kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk\") pod \"maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.453079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.453063 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlm7f\" (UniqueName: \"kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f\") pod \"maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.475842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.475813 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:00.485984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.485960 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:00.491843 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.491825 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:00.891333 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:00.891299 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bab33d3_e8b8_4489_8ffe_59795b2ecc5e.slice/crio-f72197e06d85725816e0a710e26c22ec4fce087b045302f7811384e9b33f4662 WatchSource:0}: Error finding container f72197e06d85725816e0a710e26c22ec4fce087b045302f7811384e9b33f4662: Status 404 returned error can't find the container with id f72197e06d85725816e0a710e26c22ec4fce087b045302f7811384e9b33f4662 Nov 03 00:15:00.891483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.891373 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx"] Nov 03 00:15:00.941785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.941566 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr"] Nov 03 00:15:00.944989 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:00.944795 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda15001b8_2dc0_48c1_9ec4_e83cf1f1dfb8.slice/crio-0850277d1ef4935ec5025cb99b8ef6920c8746c008c8f6e260b3f3a74af43062 WatchSource:0}: Error finding container 0850277d1ef4935ec5025cb99b8ef6920c8746c008c8f6e260b3f3a74af43062: Status 404 returned error can't find the container with id 0850277d1ef4935ec5025cb99b8ef6920c8746c008c8f6e260b3f3a74af43062 Nov 03 00:15:00.944989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:00.944912 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j"] Nov 03 00:15:00.945943 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:00.945920 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa7a606_f8ee_448e_bc24_a3bb6c51ab8b.slice/crio-df5de5c8db781475fd6cab5035e8c348426bf323d7609fe0eb20bccc2ad13bcf WatchSource:0}: Error finding container df5de5c8db781475fd6cab5035e8c348426bf323d7609fe0eb20bccc2ad13bcf: Status 404 returned error can't find the container with id df5de5c8db781475fd6cab5035e8c348426bf323d7609fe0eb20bccc2ad13bcf Nov 03 00:15:01.277154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:01.277119 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerStarted","Data":"df5de5c8db781475fd6cab5035e8c348426bf323d7609fe0eb20bccc2ad13bcf"} Nov 03 00:15:01.278640 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:01.278597 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerStarted","Data":"f72197e06d85725816e0a710e26c22ec4fce087b045302f7811384e9b33f4662"} Nov 03 00:15:01.279833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:01.279811 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerStarted","Data":"0850277d1ef4935ec5025cb99b8ef6920c8746c008c8f6e260b3f3a74af43062"} Nov 03 00:15:02.794326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.794293 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz"] Nov 03 00:15:02.799483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.799464 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:02.907572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.907538 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz"] Nov 03 00:15:02.907800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.907780 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-168f474d-dockercfg-7wngg\"" Nov 03 00:15:02.907867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.907854 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-168f474d\"" Nov 03 00:15:02.927215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.927174 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6jf4\" (UniqueName: \"kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4\") pod \"payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:02.927393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:02.927348 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:03.028605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:03.028565 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6jf4\" (UniqueName: \"kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4\") pod \"payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:03.028835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:03.028676 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:03.029316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:03.029296 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:03.087066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:03.086990 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6jf4\" (UniqueName: \"kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4\") pod \"payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:03.114043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:03.114013 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:03.497526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:03.497493 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz"] Nov 03 00:15:04.346307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:04.344923 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" event={"ID":"7ed6fca2-3abe-4395-adba-a0cca9323361","Type":"ContainerStarted","Data":"ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102"} Nov 03 00:15:04.346307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:04.344996 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" event={"ID":"7ed6fca2-3abe-4395-adba-a0cca9323361","Type":"ContainerStarted","Data":"842a3457fa51a80a0eaf471f75382cf63f006d17989daf8e74fbe47825e532f2"} Nov 03 00:15:04.346307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:04.345976 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:06.397562 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.397523 5144 generic.go:356] "Generic (PLEG): container finished" podID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerID="a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790" exitCode=0 Nov 03 00:15:06.398177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.397587 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" event={"ID":"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f","Type":"ContainerDied","Data":"a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790"} Nov 03 00:15:06.398177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.398080 5144 scope.go:122] "RemoveContainer" containerID="a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790" Nov 03 00:15:06.399614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.399584 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:06.399771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.399696 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:06.399771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.399736 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:06.585518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:06.585431 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" podStartSLOduration=4.585401377 podStartE2EDuration="4.585401377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:04.467158637 +0000 UTC m=+594.612853118" watchObservedRunningTime="2026-11-03 00:15:06.585401377 +0000 UTC m=+596.731095831" Nov 03 00:15:07.444982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:07.444909 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" event={"ID":"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f","Type":"ContainerStarted","Data":"503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9"} Nov 03 00:15:07.809657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:07.807047 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:15:07.809657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:07.808122 5144 patch_prober.go:29] interesting pod/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:15:07.809657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:07.808175 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:15:08.813009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:08.812970 5144 patch_prober.go:29] interesting pod/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:15:08.813619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:08.813583 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:15:09.135738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:09.135574 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:15:09.805284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:09.805006 5144 patch_prober.go:29] interesting pod/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:15:09.805284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:09.805061 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:15:10.179322 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:10.179292 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:15:10.186304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:10.186125 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:15:10.532892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:10.530357 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerStarted","Data":"9ef8b6bcbbbcfb3e013e3aa5ec237a21fb4ad19c7f516ed70ee23820cc51452f"} Nov 03 00:15:10.657717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:10.657413 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" podStartSLOduration=1.99852711 podStartE2EDuration="10.657391155s" totalImagesPullingTime="8.658864045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:10.653008015 +0000 UTC m=+600.798702509" watchObservedRunningTime="2026-11-03 00:15:10.657391155 +0000 UTC m=+600.803085611" Nov 03 00:15:10.803878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:10.803823 5144 patch_prober.go:29] interesting pod/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:15:10.804069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:10.803890 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:15:11.550115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.549235 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerStarted","Data":"f42d4575f68b53b3544f3a04f2709bf1a34311aa5a9a8facde30437edb2332ff"} Nov 03 00:15:11.552699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.552670 5144 generic.go:356] "Generic (PLEG): container finished" podID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerID="b76d41c9158962921afa3b94cf4cc8737c7ccaf0fa790f74aecee5a86279f5d9" exitCode=60 Nov 03 00:15:11.552844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.552749 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerDied","Data":"b76d41c9158962921afa3b94cf4cc8737c7ccaf0fa790f74aecee5a86279f5d9"} Nov 03 00:15:11.554140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.554111 5144 scope.go:122] "RemoveContainer" containerID="b76d41c9158962921afa3b94cf4cc8737c7ccaf0fa790f74aecee5a86279f5d9" Nov 03 00:15:11.554441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.554188 5144 generic.go:356] "Generic (PLEG): container finished" podID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerID="9ef8b6bcbbbcfb3e013e3aa5ec237a21fb4ad19c7f516ed70ee23820cc51452f" exitCode=60 Nov 03 00:15:11.554441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.554202 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerDied","Data":"9ef8b6bcbbbcfb3e013e3aa5ec237a21fb4ad19c7f516ed70ee23820cc51452f"} Nov 03 00:15:11.554441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.554410 5144 scope.go:122] "RemoveContainer" containerID="9ef8b6bcbbbcfb3e013e3aa5ec237a21fb4ad19c7f516ed70ee23820cc51452f" Nov 03 00:15:11.666050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.665992 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" podStartSLOduration=2.084240663 podStartE2EDuration="11.665972171s" totalImagesPullingTime="9.581731508s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:11.664201276 +0000 UTC m=+601.809895750" watchObservedRunningTime="2026-11-03 00:15:11.665972171 +0000 UTC m=+601.811666627" Nov 03 00:15:11.803943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.803908 5144 patch_prober.go:29] interesting pod/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:15:11.804110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:11.803966 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:15:12.169101 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:12.169018 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:15:12.185848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.185813 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:15:12.186014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.185883 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:15:12.563444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.563401 5144 generic.go:356] "Generic (PLEG): container finished" podID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerID="f42d4575f68b53b3544f3a04f2709bf1a34311aa5a9a8facde30437edb2332ff" exitCode=60 Nov 03 00:15:12.563950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.563482 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerDied","Data":"f42d4575f68b53b3544f3a04f2709bf1a34311aa5a9a8facde30437edb2332ff"} Nov 03 00:15:12.563950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.563806 5144 scope.go:122] "RemoveContainer" containerID="f42d4575f68b53b3544f3a04f2709bf1a34311aa5a9a8facde30437edb2332ff" Nov 03 00:15:12.566537 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.566514 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerStarted","Data":"cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2"} Nov 03 00:15:12.570706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.570681 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerStarted","Data":"b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d"} Nov 03 00:15:12.809961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.809923 5144 patch_prober.go:29] interesting pod/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Nov 03 00:15:12.810166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.809982 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Nov 03 00:15:12.925450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:12.925380 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" podStartSLOduration=3.69989825 podStartE2EDuration="12.925360137s" totalImagesPullingTime="9.225461887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:12.923462335 +0000 UTC m=+603.069156797" watchObservedRunningTime="2026-11-03 00:15:12.925360137 +0000 UTC m=+603.071054600" Nov 03 00:15:13.579025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.578991 5144 generic.go:356] "Generic (PLEG): container finished" podID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerID="cc022f952667c33d6a9c7fb32ab47ffa3b0fb6713b91c8a6f9411ddb53a73f0e" exitCode=60 Nov 03 00:15:13.579553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.579040 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerDied","Data":"cc022f952667c33d6a9c7fb32ab47ffa3b0fb6713b91c8a6f9411ddb53a73f0e"} Nov 03 00:15:13.579553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.579114 5144 scope.go:122] "RemoveContainer" containerID="f42d4575f68b53b3544f3a04f2709bf1a34311aa5a9a8facde30437edb2332ff" Nov 03 00:15:13.579553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.579513 5144 scope.go:122] "RemoveContainer" containerID="cc022f952667c33d6a9c7fb32ab47ffa3b0fb6713b91c8a6f9411ddb53a73f0e" Nov 03 00:15:13.579885 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:13.579857 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx_odh-ai-gateway-infra(5bab33d3-e8b8-4489-8ffe-59795b2ecc5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" Nov 03 00:15:13.581677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.581655 5144 generic.go:356] "Generic (PLEG): container finished" podID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerID="cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2" exitCode=60 Nov 03 00:15:13.581791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.581758 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerDied","Data":"cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2"} Nov 03 00:15:13.581935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.581915 5144 scope.go:122] "RemoveContainer" containerID="cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2" Nov 03 00:15:13.582157 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:13.582135 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr_odh-ai-gateway-infra(a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" Nov 03 00:15:13.584162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.584139 5144 generic.go:356] "Generic (PLEG): container finished" podID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerID="b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d" exitCode=60 Nov 03 00:15:13.584271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.584234 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerDied","Data":"b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d"} Nov 03 00:15:13.584427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.584411 5144 scope.go:122] "RemoveContainer" containerID="b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d" Nov 03 00:15:13.584644 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:13.584610 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-z4c6j_odh-ai-gateway-infra(cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" Nov 03 00:15:13.596998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.596968 5144 scope.go:122] "RemoveContainer" containerID="b76d41c9158962921afa3b94cf4cc8737c7ccaf0fa790f74aecee5a86279f5d9" Nov 03 00:15:13.609617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.609597 5144 scope.go:122] "RemoveContainer" containerID="9ef8b6bcbbbcfb3e013e3aa5ec237a21fb4ad19c7f516ed70ee23820cc51452f" Nov 03 00:15:13.808068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.807990 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:15:13.809126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:13.809102 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:15:14.591238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.591210 5144 generic.go:356] "Generic (PLEG): container finished" podID="2a55c316-a586-4377-99eb-164c028773cf" containerID="04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2" exitCode=0 Nov 03 00:15:14.591729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.591287 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" event={"ID":"2a55c316-a586-4377-99eb-164c028773cf","Type":"ContainerDied","Data":"04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2"} Nov 03 00:15:14.591729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.591720 5144 scope.go:122] "RemoveContainer" containerID="04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2" Nov 03 00:15:14.600255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.600212 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:14.600386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.600295 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:14.600386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.600339 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:14.600815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.600797 5144 scope.go:122] "RemoveContainer" containerID="cc022f952667c33d6a9c7fb32ab47ffa3b0fb6713b91c8a6f9411ddb53a73f0e" Nov 03 00:15:14.601010 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:14.600990 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx_odh-ai-gateway-infra(5bab33d3-e8b8-4489-8ffe-59795b2ecc5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" Nov 03 00:15:14.602198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.602181 5144 scope.go:122] "RemoveContainer" containerID="cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2" Nov 03 00:15:14.602406 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:14.602383 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr_odh-ai-gateway-infra(a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" Nov 03 00:15:14.603971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:14.603954 5144 scope.go:122] "RemoveContainer" containerID="b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d" Nov 03 00:15:14.604142 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:14.604119 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-z4c6j_odh-ai-gateway-infra(cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" Nov 03 00:15:15.620658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.614401 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" event={"ID":"2a55c316-a586-4377-99eb-164c028773cf","Type":"ContainerStarted","Data":"403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6"} Nov 03 00:15:15.634363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.633243 5144 generic.go:356] "Generic (PLEG): container finished" podID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerID="f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d" exitCode=0 Nov 03 00:15:15.634363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.633332 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" event={"ID":"c6dc027a-bd51-43db-91a8-5cca7787a86b","Type":"ContainerDied","Data":"f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d"} Nov 03 00:15:15.634363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.633827 5144 scope.go:122] "RemoveContainer" containerID="f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d" Nov 03 00:15:15.666285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.664543 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:15.666285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.664678 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:15.666285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:15.664737 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:16.401324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.401294 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:16.496107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.496075 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:15:16.496350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.496326 5144 patch_prober.go:29] interesting pod/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:15:16.496442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.496379 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 00:15:16.678249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.678161 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" event={"ID":"c6dc027a-bd51-43db-91a8-5cca7787a86b","Type":"ContainerStarted","Data":"aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33"} Nov 03 00:15:16.885589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.885546 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:15:16.885871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.885848 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:15:16.885969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:16.885902 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 00:15:17.496620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:17.496571 5144 patch_prober.go:29] interesting pod/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:15:17.496866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:17.496671 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 00:15:17.885906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:17.885869 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:15:17.886303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:17.885920 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 00:15:18.496335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:18.496299 5144 patch_prober.go:29] interesting pod/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:15:18.496514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:18.496352 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 00:15:18.885955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:18.885919 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:15:18.886346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:18.885975 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 00:15:19.497070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:19.497035 5144 patch_prober.go:29] interesting pod/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:15:19.497372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:19.497341 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 00:15:19.885510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:19.885474 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:15:19.885709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:19.885541 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 00:15:20.496950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:20.496915 5144 patch_prober.go:29] interesting pod/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body= Nov 03 00:15:20.497332 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:20.496977 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" Nov 03 00:15:20.885800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:20.885763 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 00:15:20.885975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:20.885818 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 00:15:21.500652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:21.500611 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:15:21.501107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:21.501088 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:15:21.501782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:21.501760 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:15:21.968235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:21.968202 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:15:21.968889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:21.968853 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:15:21.969704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:21.969679 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:15:24.989497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:24.989459 5144 scope.go:122] "RemoveContainer" containerID="b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d" Nov 03 00:15:25.753560 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:25.753530 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerStarted","Data":"e83b4069d1c6fad9f351a785b461a2363793c5c09737f31f626a46cb207a486d"} Nov 03 00:15:26.762328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:26.762297 5144 generic.go:356] "Generic (PLEG): container finished" podID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerID="e83b4069d1c6fad9f351a785b461a2363793c5c09737f31f626a46cb207a486d" exitCode=60 Nov 03 00:15:26.762878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:26.762368 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerDied","Data":"e83b4069d1c6fad9f351a785b461a2363793c5c09737f31f626a46cb207a486d"} Nov 03 00:15:26.762878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:26.762431 5144 scope.go:122] "RemoveContainer" containerID="b550d2f3695201db83016aff04cc738c2342bba2282561b0e285427cfc619f1d" Nov 03 00:15:26.762878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:26.762814 5144 scope.go:122] "RemoveContainer" containerID="e83b4069d1c6fad9f351a785b461a2363793c5c09737f31f626a46cb207a486d" Nov 03 00:15:26.763113 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:26.763091 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894415-z4c6j_odh-ai-gateway-infra(cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" Nov 03 00:15:26.887370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:26.887332 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j"] Nov 03 00:15:26.988797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:26.988767 5144 scope.go:122] "RemoveContainer" containerID="cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2" Nov 03 00:15:27.042600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.042569 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58"] Nov 03 00:15:27.046128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.046105 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.101533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.101474 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:27.104684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.104657 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58"] Nov 03 00:15:27.166321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.166293 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.166462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.166328 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.166686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.166659 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9thzc\" (UniqueName: \"kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.267794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.267758 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9thzc\" (UniqueName: \"kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.267976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.267887 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.267976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.267914 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.268685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.268666 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.270357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.270332 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.339716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.333834 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9thzc\" (UniqueName: \"kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc\") pod \"maas-api-revoke-keys-e2e-ait-01450ec4-98c58\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.357391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.357366 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:27.740707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.740099 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58"] Nov 03 00:15:27.748346 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:27.748321 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5938d3e5_8b79_4841_a954_9605472ef1ce.slice/crio-0b9c62d78352766f15129b569ae07cd7bee5eb718ee40e338b96f18ccc64c6f0 WatchSource:0}: Error finding container 0b9c62d78352766f15129b569ae07cd7bee5eb718ee40e338b96f18ccc64c6f0: Status 404 returned error can't find the container with id 0b9c62d78352766f15129b569ae07cd7bee5eb718ee40e338b96f18ccc64c6f0 Nov 03 00:15:27.770575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.770544 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerStarted","Data":"60fbaabb9cf695aa3f28885102d37b018f843e2fe5ffcba89fe9829149c90112"} Nov 03 00:15:27.775814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.775782 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" event={"ID":"5938d3e5-8b79-4841-a954-9605472ef1ce","Type":"ContainerStarted","Data":"0b9c62d78352766f15129b569ae07cd7bee5eb718ee40e338b96f18ccc64c6f0"} Nov 03 00:15:27.965194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.965163 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:27.989026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:27.989005 5144 scope.go:122] "RemoveContainer" containerID="cc022f952667c33d6a9c7fb32ab47ffa3b0fb6713b91c8a6f9411ddb53a73f0e" Nov 03 00:15:28.076690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.076528 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca\") pod \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " Nov 03 00:15:28.076690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.076661 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827\" (UniqueName: \"kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827\") pod \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\" (UID: \"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b\") " Nov 03 00:15:28.076952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.076926 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca" pod "cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" (UID: "cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:28.078874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.078849 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827" pod "cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" (UID: "cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b"). InnerVolumeSpecName "kube-api-access-vn827". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:28.177915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.177887 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.177915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.177915 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vn827\" (UniqueName: \"kubernetes.io/projected/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b-kube-api-access-vn827\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:28.784420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.784388 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerStarted","Data":"00843e7bb4f1c4af6d7882de23769db7bdfe2a9d063b466ba105c384d352830a"} Nov 03 00:15:28.786571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.786548 5144 generic.go:356] "Generic (PLEG): container finished" podID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerID="60fbaabb9cf695aa3f28885102d37b018f843e2fe5ffcba89fe9829149c90112" exitCode=60 Nov 03 00:15:28.786717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.786653 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerDied","Data":"60fbaabb9cf695aa3f28885102d37b018f843e2fe5ffcba89fe9829149c90112"} Nov 03 00:15:28.786790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.786719 5144 scope.go:122] "RemoveContainer" containerID="cf18c00568e3253c6f38d643d4e65bdafc25f3c50fb0386f00b188166b323ac2" Nov 03 00:15:28.787074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.787051 5144 scope.go:122] "RemoveContainer" containerID="60fbaabb9cf695aa3f28885102d37b018f843e2fe5ffcba89fe9829149c90112" Nov 03 00:15:28.787369 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:28.787337 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr_odh-ai-gateway-infra(a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" Nov 03 00:15:28.789932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.789913 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" Nov 03 00:15:28.790020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.789943 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j" event={"ID":"cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b","Type":"ContainerDied","Data":"df5de5c8db781475fd6cab5035e8c348426bf323d7609fe0eb20bccc2ad13bcf"} Nov 03 00:15:28.791660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.791599 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" event={"ID":"5938d3e5-8b79-4841-a954-9605472ef1ce","Type":"ContainerStarted","Data":"fee665cc55f351c2dbed3955ca2430a200d8818649d46f746bd127d19ef33f00"} Nov 03 00:15:28.856089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.856061 5144 scope.go:122] "RemoveContainer" containerID="e83b4069d1c6fad9f351a785b461a2363793c5c09737f31f626a46cb207a486d" Nov 03 00:15:28.907231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.907204 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr"] Nov 03 00:15:28.952421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.952388 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs"] Nov 03 00:15:28.952932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.952917 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.953004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.952933 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.953004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.952944 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.953004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.952949 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.953100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.953047 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.953100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.953056 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.953100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.953062 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:28.956080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:28.956060 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.008284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.008249 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j"] Nov 03 00:15:29.011761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.011739 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894415-z4c6j"] Nov 03 00:15:29.048194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.048109 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs"] Nov 03 00:15:29.087054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.087029 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.087201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.087179 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.087338 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.087321 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qzc4\" (UniqueName: \"kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.187867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.187841 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.188015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.187959 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qzc4\" (UniqueName: \"kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.188082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.188012 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.188744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.188719 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.190314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.190292 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.244474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.244449 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qzc4\" (UniqueName: \"kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4\") pod \"maas-api-revoke-keys-e2e-ait-12537192-7q5gs\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.269151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.269109 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:29.404500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.404469 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx"] Nov 03 00:15:29.666553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.666486 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs"] Nov 03 00:15:29.667774 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:29.667748 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39fc68b_4ea4_40d8_9b40_cc9dda8ef6d7.slice/crio-0a8e4544cb8935b39e7d7ab248728fb2a6737a1e05b6c47165e87d1c43a662ed WatchSource:0}: Error finding container 0a8e4544cb8935b39e7d7ab248728fb2a6737a1e05b6c47165e87d1c43a662ed: Status 404 returned error can't find the container with id 0a8e4544cb8935b39e7d7ab248728fb2a6737a1e05b6c47165e87d1c43a662ed Nov 03 00:15:29.802026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.801990 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" event={"ID":"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7","Type":"ContainerStarted","Data":"0a8e4544cb8935b39e7d7ab248728fb2a6737a1e05b6c47165e87d1c43a662ed"} Nov 03 00:15:29.803794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.803767 5144 generic.go:356] "Generic (PLEG): container finished" podID="5938d3e5-8b79-4841-a954-9605472ef1ce" containerID="fee665cc55f351c2dbed3955ca2430a200d8818649d46f746bd127d19ef33f00" exitCode=0 Nov 03 00:15:29.803979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.803855 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" event={"ID":"5938d3e5-8b79-4841-a954-9605472ef1ce","Type":"ContainerDied","Data":"fee665cc55f351c2dbed3955ca2430a200d8818649d46f746bd127d19ef33f00"} Nov 03 00:15:29.807835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.806752 5144 generic.go:356] "Generic (PLEG): container finished" podID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerID="00843e7bb4f1c4af6d7882de23769db7bdfe2a9d063b466ba105c384d352830a" exitCode=60 Nov 03 00:15:29.807835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.806893 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerDied","Data":"00843e7bb4f1c4af6d7882de23769db7bdfe2a9d063b466ba105c384d352830a"} Nov 03 00:15:29.807835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:29.806953 5144 scope.go:122] "RemoveContainer" containerID="cc022f952667c33d6a9c7fb32ab47ffa3b0fb6713b91c8a6f9411ddb53a73f0e" Nov 03 00:15:30.008484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.008441 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" path="/var/lib/kubelet/pods/cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b/volumes" Nov 03 00:15:30.111339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.108459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:30.189823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.189347 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:30.192241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.192219 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:30.204196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204164 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca\") pod \"5938d3e5-8b79-4841-a954-9605472ef1ce\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " Nov 03 00:15:30.204324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204252 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk\" (UniqueName: \"kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk\") pod \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " Nov 03 00:15:30.204324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204312 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f\" (UniqueName: \"kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f\") pod \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " Nov 03 00:15:30.204431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204399 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca\") pod \"5938d3e5-8b79-4841-a954-9605472ef1ce\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " Nov 03 00:15:30.204480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204448 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc\" (UniqueName: \"kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc\") pod \"5938d3e5-8b79-4841-a954-9605472ef1ce\" (UID: \"5938d3e5-8b79-4841-a954-9605472ef1ce\") " Nov 03 00:15:30.204532 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204508 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca\") pod \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\" (UID: \"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8\") " Nov 03 00:15:30.204582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.204543 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca\") pod \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\" (UID: \"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e\") " Nov 03 00:15:30.205336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.205310 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca" pod "5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" (UID: "5bab33d3-e8b8-4489-8ffe-59795b2ecc5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:30.205739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.205338 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca" pod "a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" (UID: "a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:30.205930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.205614 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca" pod "5938d3e5-8b79-4841-a954-9605472ef1ce" (UID: "5938d3e5-8b79-4841-a954-9605472ef1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:30.207977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.207842 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca" pod "5938d3e5-8b79-4841-a954-9605472ef1ce" (UID: "5938d3e5-8b79-4841-a954-9605472ef1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:30.208095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.208058 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk" pod "5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" (UID: "5bab33d3-e8b8-4489-8ffe-59795b2ecc5e"). InnerVolumeSpecName "kube-api-access-mpkmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:30.208283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.208251 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f" pod "a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" (UID: "a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8"). InnerVolumeSpecName "kube-api-access-nlm7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:30.208812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.208793 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc" pod "5938d3e5-8b79-4841-a954-9605472ef1ce" (UID: "5938d3e5-8b79-4841-a954-9605472ef1ce"). InnerVolumeSpecName "kube-api-access-9thzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:30.305862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305834 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.305862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305859 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpkmk\" (UniqueName: \"kubernetes.io/projected/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-kube-api-access-mpkmk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.306020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305870 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlm7f\" (UniqueName: \"kubernetes.io/projected/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-kube-api-access-nlm7f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.306020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305879 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5938d3e5-8b79-4841-a954-9605472ef1ce-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.306020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305888 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9thzc\" (UniqueName: \"kubernetes.io/projected/5938d3e5-8b79-4841-a954-9605472ef1ce-kube-api-access-9thzc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.306020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305896 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.306020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.305904 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:30.862866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.862781 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" Nov 03 00:15:30.862866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.862789 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx" event={"ID":"5bab33d3-e8b8-4489-8ffe-59795b2ecc5e","Type":"ContainerDied","Data":"f72197e06d85725816e0a710e26c22ec4fce087b045302f7811384e9b33f4662"} Nov 03 00:15:30.863372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.862879 5144 scope.go:122] "RemoveContainer" containerID="00843e7bb4f1c4af6d7882de23769db7bdfe2a9d063b466ba105c384d352830a" Nov 03 00:15:30.864518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.864498 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" event={"ID":"a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8","Type":"ContainerDied","Data":"0850277d1ef4935ec5025cb99b8ef6920c8746c008c8f6e260b3f3a74af43062"} Nov 03 00:15:30.864595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.864587 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr" Nov 03 00:15:30.873061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.873036 5144 generic.go:356] "Generic (PLEG): container finished" podID="f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" containerID="2fdac7935ceb1aba0b1f16261d8efa54f537f78e7a2ee69924daae9034e97cab" exitCode=0 Nov 03 00:15:30.873172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.873141 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" event={"ID":"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7","Type":"ContainerDied","Data":"2fdac7935ceb1aba0b1f16261d8efa54f537f78e7a2ee69924daae9034e97cab"} Nov 03 00:15:30.874592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.874520 5144 scope.go:122] "RemoveContainer" containerID="60fbaabb9cf695aa3f28885102d37b018f843e2fe5ffcba89fe9829149c90112" Nov 03 00:15:30.874701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.874671 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" event={"ID":"5938d3e5-8b79-4841-a954-9605472ef1ce","Type":"ContainerDied","Data":"0b9c62d78352766f15129b569ae07cd7bee5eb718ee40e338b96f18ccc64c6f0"} Nov 03 00:15:30.874759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.874710 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9c62d78352766f15129b569ae07cd7bee5eb718ee40e338b96f18ccc64c6f0" Nov 03 00:15:30.874759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:30.874750 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58" Nov 03 00:15:31.025030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:31.025004 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr"] Nov 03 00:15:31.028320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:31.028297 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-2ccce380-29894415-4cnrr"] Nov 03 00:15:31.187788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:31.187717 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx"] Nov 03 00:15:31.192357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:31.192328 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-01450ec4-29894415-tl7lx"] Nov 03 00:15:31.998382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:31.998347 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" path="/var/lib/kubelet/pods/5bab33d3-e8b8-4489-8ffe-59795b2ecc5e/volumes" Nov 03 00:15:31.998912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:31.998802 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" path="/var/lib/kubelet/pods/a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8/volumes" Nov 03 00:15:32.059837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.059807 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:32.125218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.125185 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca\") pod \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " Nov 03 00:15:32.125371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.125239 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4\" (UniqueName: \"kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4\") pod \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " Nov 03 00:15:32.125371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.125311 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca\") pod \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\" (UID: \"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7\") " Nov 03 00:15:32.125703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.125674 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca" pod "f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" (UID: "f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:32.127712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.127650 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca" pod "f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" (UID: "f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:32.127819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.127705 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4" pod "f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" (UID: "f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7"). InnerVolumeSpecName "kube-api-access-8qzc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:32.227177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.227136 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:32.227177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.227169 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qzc4\" (UniqueName: \"kubernetes.io/projected/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-kube-api-access-8qzc4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:32.227177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.227181 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:32.891479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.891448 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" Nov 03 00:15:32.891702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.891478 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs" event={"ID":"f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7","Type":"ContainerDied","Data":"0a8e4544cb8935b39e7d7ab248728fb2a6737a1e05b6c47165e87d1c43a662ed"} Nov 03 00:15:32.891702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:32.891527 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a8e4544cb8935b39e7d7ab248728fb2a6737a1e05b6c47165e87d1c43a662ed" Nov 03 00:15:33.903588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:33.903495 5144 generic.go:356] "Generic (PLEG): container finished" podID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerID="eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd" exitCode=0 Nov 03 00:15:33.903588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:33.903553 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" event={"ID":"3ae357a1-8b3d-4210-b7f3-0edb0b961106","Type":"ContainerDied","Data":"eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd"} Nov 03 00:15:33.904088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:33.903980 5144 scope.go:122] "RemoveContainer" containerID="eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd" Nov 03 00:15:33.905669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:33.905621 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:33.905797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:33.905729 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:33.905797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:33.905778 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:15:34.911463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:34.911424 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" event={"ID":"3ae357a1-8b3d-4210-b7f3-0edb0b961106","Type":"ContainerStarted","Data":"e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e"} Nov 03 00:15:35.739337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.739250 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:15:35.739789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.739757 5144 patch_prober.go:29] interesting pod/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:15:35.739937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.739816 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 00:15:35.975102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975071 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8"] Nov 03 00:15:35.975576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975562 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.975619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975577 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.975619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975587 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" containerName="revoke-keys" Nov 03 00:15:35.975619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975592 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" containerName="revoke-keys" Nov 03 00:15:35.975619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975605 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.975619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975613 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975662 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975668 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfa7a606-f8ee-448e-bc24-a3bb6c51ab8b" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975682 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975690 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975701 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975706 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975716 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5938d3e5-8b79-4841-a954-9605472ef1ce" containerName="revoke-keys" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975721 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5938d3e5-8b79-4841-a954-9605472ef1ce" containerName="revoke-keys" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975731 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.975818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975736 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975826 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975832 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975841 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" containerName="revoke-keys" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975849 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975861 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975866 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:35.976093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.975872 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5938d3e5-8b79-4841-a954-9605472ef1ce" containerName="revoke-keys" Nov 03 00:15:35.978082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:35.978064 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.039080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.039051 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:15:36.039389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.039370 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:36.042999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.042975 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8"] Nov 03 00:15:36.067515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.067480 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmm7t\" (UniqueName: \"kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.067748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.067726 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.067799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.067760 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.169132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.169096 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.169132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.169148 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.169407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.169231 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmm7t\" (UniqueName: \"kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.169849 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.169824 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.171690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.171667 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.235657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.232244 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmm7t\" (UniqueName: \"kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t\") pod \"maas-api-revoke-keys-e2e-ait-2ccce380-nxms8\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.290357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.290276 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:36.681969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.681942 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8"] Nov 03 00:15:36.684089 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:36.684056 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0fb5b4_0d47_4076_b3ea_8e8d40b94d0f.slice/crio-eec9d67e1f0f50efdcdd35162ceade22864b7de85df3b0836d4f430a2ebc64a1 WatchSource:0}: Error finding container eec9d67e1f0f50efdcdd35162ceade22864b7de85df3b0836d4f430a2ebc64a1: Status 404 returned error can't find the container with id eec9d67e1f0f50efdcdd35162ceade22864b7de85df3b0836d4f430a2ebc64a1 Nov 03 00:15:36.739732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.739058 5144 patch_prober.go:29] interesting pod/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:15:36.739732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.739119 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 00:15:36.950584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:36.950551 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" event={"ID":"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f","Type":"ContainerStarted","Data":"eec9d67e1f0f50efdcdd35162ceade22864b7de85df3b0836d4f430a2ebc64a1"} Nov 03 00:15:37.737310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:37.736889 5144 patch_prober.go:29] interesting pod/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:15:37.737310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:37.736948 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 00:15:37.957939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:37.957903 5144 generic.go:356] "Generic (PLEG): container finished" podID="8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" containerID="bfa099a276123f59e87dc9530977247541c157ca86ccecd559abe62bc57ed811" exitCode=0 Nov 03 00:15:37.958082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:37.957993 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" event={"ID":"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f","Type":"ContainerDied","Data":"bfa099a276123f59e87dc9530977247541c157ca86ccecd559abe62bc57ed811"} Nov 03 00:15:38.740674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:38.735994 5144 patch_prober.go:29] interesting pod/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:15:38.740674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:38.736052 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 00:15:39.178049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.178016 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:39.200129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.200094 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca\") pod \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " Nov 03 00:15:39.200290 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.200197 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t\" (UniqueName: \"kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t\") pod \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " Nov 03 00:15:39.200358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.200340 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca\") pod \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\" (UID: \"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f\") " Nov 03 00:15:39.200498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.200475 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca" pod "8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" (UID: "8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:39.200727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.200701 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:39.202665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.202619 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca" pod "8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" (UID: "8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:39.202811 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.202788 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t" pod "8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" (UID: "8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f"). InnerVolumeSpecName "kube-api-access-lmm7t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:39.301436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.301351 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:39.301436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.301387 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmm7t\" (UniqueName: \"kubernetes.io/projected/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f-kube-api-access-lmm7t\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:39.738715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.734216 5144 patch_prober.go:29] interesting pod/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 00:15:39.738715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.734293 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 00:15:39.972994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.972963 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" Nov 03 00:15:39.972994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.972987 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8" event={"ID":"8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f","Type":"ContainerDied","Data":"eec9d67e1f0f50efdcdd35162ceade22864b7de85df3b0836d4f430a2ebc64a1"} Nov 03 00:15:39.973451 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:39.973038 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec9d67e1f0f50efdcdd35162ceade22864b7de85df3b0836d4f430a2ebc64a1" Nov 03 00:15:40.137657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:40.134895 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:15:40.738436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:40.738407 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:15:40.980175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:40.980144 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:15:42.155193 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:42.155147 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:15:42.379572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.379537 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2"] Nov 03 00:15:42.380060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.380046 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:42.380104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.380061 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bab33d3-e8b8-4489-8ffe-59795b2ecc5e" containerName="cleanup" Nov 03 00:15:42.380104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.380077 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" containerName="revoke-keys" Nov 03 00:15:42.380104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.380082 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" containerName="revoke-keys" Nov 03 00:15:42.380200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.380152 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" containerName="revoke-keys" Nov 03 00:15:42.380200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.380164 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a15001b8-2dc0-48c1-9ec4-e83cf1f1dfb8" containerName="cleanup" Nov 03 00:15:42.382394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.382378 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.435694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.435574 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:15:42.438068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.438049 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfb64\" (UniqueName: \"kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.438127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.438115 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.438256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.438242 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.488445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.488412 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2"] Nov 03 00:15:42.535684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.535602 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:42.539146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.539122 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.539217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.539164 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfb64\" (UniqueName: \"kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.539253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.539214 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.539848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.539826 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.541596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.541570 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.596346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.596314 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfb64\" (UniqueName: \"kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64\") pod \"maas-api-revoke-keys-e2e-derive-168f474d-56kv2\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:42.695015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:42.694921 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:43.075254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:43.075224 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2"] Nov 03 00:15:43.076428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:43.076404 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7352f42c_44ed_4ca9_a94c_25d8c7022d58.slice/crio-2555e6fca33e8799725b71a8e14b2fa20ab502e435c6ea72ac4b7034d5510d67 WatchSource:0}: Error finding container 2555e6fca33e8799725b71a8e14b2fa20ab502e435c6ea72ac4b7034d5510d67: Status 404 returned error can't find the container with id 2555e6fca33e8799725b71a8e14b2fa20ab502e435c6ea72ac4b7034d5510d67 Nov 03 00:15:44.004747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:44.004697 5144 generic.go:356] "Generic (PLEG): container finished" podID="7352f42c-44ed-4ca9-a94c-25d8c7022d58" containerID="16af69988d7f80ee40ebadbf7f3f558f38a3318ee7534d51092714266ae83b87" exitCode=0 Nov 03 00:15:44.005178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:44.004842 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" event={"ID":"7352f42c-44ed-4ca9-a94c-25d8c7022d58","Type":"ContainerDied","Data":"16af69988d7f80ee40ebadbf7f3f558f38a3318ee7534d51092714266ae83b87"} Nov 03 00:15:44.005178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:44.004888 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" event={"ID":"7352f42c-44ed-4ca9-a94c-25d8c7022d58","Type":"ContainerStarted","Data":"2555e6fca33e8799725b71a8e14b2fa20ab502e435c6ea72ac4b7034d5510d67"} Nov 03 00:15:45.178611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.178589 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:45.267361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.267333 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64\" (UniqueName: \"kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64\") pod \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " Nov 03 00:15:45.267494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.267435 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca\") pod \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " Nov 03 00:15:45.267559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.267543 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca\") pod \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\" (UID: \"7352f42c-44ed-4ca9-a94c-25d8c7022d58\") " Nov 03 00:15:45.267808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.267786 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca" pod "7352f42c-44ed-4ca9-a94c-25d8c7022d58" (UID: "7352f42c-44ed-4ca9-a94c-25d8c7022d58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:45.267943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.267925 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:45.269573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.269547 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca" pod "7352f42c-44ed-4ca9-a94c-25d8c7022d58" (UID: "7352f42c-44ed-4ca9-a94c-25d8c7022d58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:45.269668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.269574 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64" pod "7352f42c-44ed-4ca9-a94c-25d8c7022d58" (UID: "7352f42c-44ed-4ca9-a94c-25d8c7022d58"). InnerVolumeSpecName "kube-api-access-xfb64". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:45.369068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.368995 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfb64\" (UniqueName: \"kubernetes.io/projected/7352f42c-44ed-4ca9-a94c-25d8c7022d58-kube-api-access-xfb64\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:45.369068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:45.369021 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7352f42c-44ed-4ca9-a94c-25d8c7022d58-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:46.035079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:46.035054 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" Nov 03 00:15:46.035233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:46.035092 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2" event={"ID":"7352f42c-44ed-4ca9-a94c-25d8c7022d58","Type":"ContainerDied","Data":"2555e6fca33e8799725b71a8e14b2fa20ab502e435c6ea72ac4b7034d5510d67"} Nov 03 00:15:46.035233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:46.035133 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2555e6fca33e8799725b71a8e14b2fa20ab502e435c6ea72ac4b7034d5510d67" Nov 03 00:15:46.774254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:46.774226 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58"] Nov 03 00:15:46.843007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:46.842969 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-01450ec4-98c58"] Nov 03 00:15:48.003071 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.003036 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5938d3e5-8b79-4841-a954-9605472ef1ce" path="/var/lib/kubelet/pods/5938d3e5-8b79-4841-a954-9605472ef1ce/volumes" Nov 03 00:15:48.234952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.234918 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs"] Nov 03 00:15:48.284161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.284085 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-12537192-7q5gs"] Nov 03 00:15:48.286364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.286341 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24"] Nov 03 00:15:48.286652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.286610 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" containerID="cri-o://503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9" gracePeriod=30 Nov 03 00:15:48.650229 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.650159 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt"] Nov 03 00:15:48.650661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.650622 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7352f42c-44ed-4ca9-a94c-25d8c7022d58" containerName="revoke-keys" Nov 03 00:15:48.650715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.650663 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7352f42c-44ed-4ca9-a94c-25d8c7022d58" containerName="revoke-keys" Nov 03 00:15:48.650787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.650778 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7352f42c-44ed-4ca9-a94c-25d8c7022d58" containerName="revoke-keys" Nov 03 00:15:48.653075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.653057 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.703833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.703807 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.703995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.703851 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlcbg\" (UniqueName: \"kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.703995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.703943 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.710450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.710241 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:15:48.710450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.710419 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:15:48.713323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.713301 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74"] Nov 03 00:15:48.713686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.713659 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" containerID="cri-o://aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33" gracePeriod=30 Nov 03 00:15:48.719043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.719019 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt"] Nov 03 00:15:48.804612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.804586 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.804773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.804721 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.804773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.804744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlcbg\" (UniqueName: \"kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.805170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.805152 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.807162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.807137 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.862772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.862738 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlcbg\" (UniqueName: \"kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg\") pod \"maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:48.965586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:48.965505 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:49.344256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:49.344232 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt"] Nov 03 00:15:49.345450 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:49.345423 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766cf582_ff64_4c0e_8276_30fcc749c761.slice/crio-c366c702a7a9e9ba8da9d84d0407f358c9f7970e00b51e12a6062a00a9d0ea01 WatchSource:0}: Error finding container c366c702a7a9e9ba8da9d84d0407f358c9f7970e00b51e12a6062a00a9d0ea01: Status 404 returned error can't find the container with id c366c702a7a9e9ba8da9d84d0407f358c9f7970e00b51e12a6062a00a9d0ea01 Nov 03 00:15:49.996668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:49.996567 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7" path="/var/lib/kubelet/pods/f39fc68b-4ea4-40d8-9b40-cc9dda8ef6d7/volumes" Nov 03 00:15:50.074139 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:50.074096 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" event={"ID":"766cf582-ff64-4c0e-8276-30fcc749c761","Type":"ContainerStarted","Data":"5e99d9e05af101034032d3e5cf21d8bf23e00137d03635d94c743583a4da8551"} Nov 03 00:15:50.074315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:50.074194 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" event={"ID":"766cf582-ff64-4c0e-8276-30fcc749c761","Type":"ContainerStarted","Data":"c366c702a7a9e9ba8da9d84d0407f358c9f7970e00b51e12a6062a00a9d0ea01"} Nov 03 00:15:50.191836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:50.191786 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" podStartSLOduration=2.191772439 podStartE2EDuration="2.191772439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:50.190109722 +0000 UTC m=+640.335804204" watchObservedRunningTime="2026-11-03 00:15:50.191772439 +0000 UTC m=+640.337466911" Nov 03 00:15:51.081893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:51.081857 5144 generic.go:356] "Generic (PLEG): container finished" podID="766cf582-ff64-4c0e-8276-30fcc749c761" containerID="5e99d9e05af101034032d3e5cf21d8bf23e00137d03635d94c743583a4da8551" exitCode=0 Nov 03 00:15:51.082307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:51.081927 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" event={"ID":"766cf582-ff64-4c0e-8276-30fcc749c761","Type":"ContainerDied","Data":"5e99d9e05af101034032d3e5cf21d8bf23e00137d03635d94c743583a4da8551"} Nov 03 00:15:51.971182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:51.971148 5144 patch_prober.go:29] interesting pod/e2e-ait-12537192-openshift-default-546d597f7-h6h74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:15:51.971364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:51.971204 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:15:52.234073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.234024 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:52.341144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.341114 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca\") pod \"766cf582-ff64-4c0e-8276-30fcc749c761\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " Nov 03 00:15:52.341283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.341194 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg\" (UniqueName: \"kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg\") pod \"766cf582-ff64-4c0e-8276-30fcc749c761\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " Nov 03 00:15:52.341283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.341215 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca\") pod \"766cf582-ff64-4c0e-8276-30fcc749c761\" (UID: \"766cf582-ff64-4c0e-8276-30fcc749c761\") " Nov 03 00:15:52.341620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.341592 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca" pod "766cf582-ff64-4c0e-8276-30fcc749c761" (UID: "766cf582-ff64-4c0e-8276-30fcc749c761"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:52.343266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.343241 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca" pod "766cf582-ff64-4c0e-8276-30fcc749c761" (UID: "766cf582-ff64-4c0e-8276-30fcc749c761"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:15:52.343363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.343342 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg" pod "766cf582-ff64-4c0e-8276-30fcc749c761" (UID: "766cf582-ff64-4c0e-8276-30fcc749c761"). InnerVolumeSpecName "kube-api-access-dlcbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:52.442499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.442467 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.442499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.442496 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlcbg\" (UniqueName: \"kubernetes.io/projected/766cf582-ff64-4c0e-8276-30fcc749c761-kube-api-access-dlcbg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:52.442690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:52.442506 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766cf582-ff64-4c0e-8276-30fcc749c761-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.096421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.096393 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" Nov 03 00:15:53.096596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.096432 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt" event={"ID":"766cf582-ff64-4c0e-8276-30fcc749c761","Type":"ContainerDied","Data":"c366c702a7a9e9ba8da9d84d0407f358c9f7970e00b51e12a6062a00a9d0ea01"} Nov 03 00:15:53.096596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.096487 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c366c702a7a9e9ba8da9d84d0407f358c9f7970e00b51e12a6062a00a9d0ea01" Nov 03 00:15:53.539132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.539110 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:15:53.655891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.655835 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.655891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.655884 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.655906 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.655927 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.655954 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656002 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656029 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656063 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656105 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs\") pod \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\" (UID: \"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f\") " Nov 03 00:15:53.656375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656162 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:53.656375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656361 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:53.656553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656381 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:53.656553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656414 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:53.656553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656472 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.656553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656486 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.656553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656495 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.656977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.656655 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:53.658346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.658324 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:53.658443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.658351 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:53.658443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.658387 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "kube-api-access-w5f6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:53.658552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.658440 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo" pod "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" (UID: "72919027-3c7d-4b04-8fd1-5f92fcaf1b4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:15:53.757423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.757398 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.757423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.757424 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.757558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.757433 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.757558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.757442 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.757558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.757451 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.757558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.757459 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5f6z\" (UniqueName: \"kubernetes.io/projected/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f-kube-api-access-w5f6z\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:53.952260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:53.952235 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:15:54.060687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060656 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.060836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060751 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.060836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060800 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.060954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060835 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.060954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060865 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.060954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060896 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.060954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.060927 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.061576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.061502 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:54.061739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.061571 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:54.061739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.061603 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:54.061739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.061664 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.061928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.061752 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy\") pod \"c6dc027a-bd51-43db-91a8-5cca7787a86b\" (UID: \"c6dc027a-bd51-43db-91a8-5cca7787a86b\") " Nov 03 00:15:54.062610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.062077 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:54.062610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.062440 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz"] Nov 03 00:15:54.062610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.062600 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:54.063624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.063586 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.063741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.063647 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6dc027a-bd51-43db-91a8-5cca7787a86b-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.063741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.063663 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.063741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.063678 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.063741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.063697 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.063959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.063902 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "kube-api-access-kgjsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:54.064235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.064209 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:54.065022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.064922 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:15:54.066974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.065179 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy" pod "c6dc027a-bd51-43db-91a8-5cca7787a86b" (UID: "c6dc027a-bd51-43db-91a8-5cca7787a86b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:15:54.109905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.109869 5144 generic.go:356] "Generic (PLEG): container finished" podID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerID="aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33" exitCode=0 Nov 03 00:15:54.110074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.109926 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" Nov 03 00:15:54.110074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.109957 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" event={"ID":"c6dc027a-bd51-43db-91a8-5cca7787a86b","Type":"ContainerDied","Data":"aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33"} Nov 03 00:15:54.110074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.110012 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74" event={"ID":"c6dc027a-bd51-43db-91a8-5cca7787a86b","Type":"ContainerDied","Data":"97e79f2727540300fddc47e89cda6cae844f7d646e1bd36659db22b4a37c8285"} Nov 03 00:15:54.110074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.110041 5144 scope.go:122] "RemoveContainer" containerID="aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33" Nov 03 00:15:54.112574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.112545 5144 generic.go:356] "Generic (PLEG): container finished" podID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerID="503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9" exitCode=0 Nov 03 00:15:54.112769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.112614 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" Nov 03 00:15:54.112769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.112662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" event={"ID":"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f","Type":"ContainerDied","Data":"503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9"} Nov 03 00:15:54.112769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.112716 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24" event={"ID":"72919027-3c7d-4b04-8fd1-5f92fcaf1b4f","Type":"ContainerDied","Data":"c93facfd545d62a44cb5becf94816ad771df4622ae9130f718bd476787616c1d"} Nov 03 00:15:54.122650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.122618 5144 scope.go:122] "RemoveContainer" containerID="f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d" Nov 03 00:15:54.131686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.131670 5144 scope.go:122] "RemoveContainer" containerID="aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33" Nov 03 00:15:54.131930 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:54.131914 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33\": container with ID starting with aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33 not found: ID does not exist" containerID="aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33" Nov 03 00:15:54.131980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.131938 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33"} err="failed to get container status \"aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33\": rpc error: code = NotFound desc = could not find container \"aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33\": container with ID starting with aed0e5b25fd1f6aaf663fe702eb00e5753d311a16725d9a821367ba0da7a6b33 not found: ID does not exist" Nov 03 00:15:54.131980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.131953 5144 scope.go:122] "RemoveContainer" containerID="f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d" Nov 03 00:15:54.132161 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:54.132147 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d\": container with ID starting with f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d not found: ID does not exist" containerID="f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d" Nov 03 00:15:54.132201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.132164 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d"} err="failed to get container status \"f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d\": rpc error: code = NotFound desc = could not find container \"f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d\": container with ID starting with f3fe197cb142249e8d3ba25139f108cb69247bc7ac98f21cf24dfd4c5c0b560d not found: ID does not exist" Nov 03 00:15:54.132201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.132175 5144 scope.go:122] "RemoveContainer" containerID="503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9" Nov 03 00:15:54.140764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.140746 5144 scope.go:122] "RemoveContainer" containerID="a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790" Nov 03 00:15:54.148732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.148716 5144 scope.go:122] "RemoveContainer" containerID="503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9" Nov 03 00:15:54.148970 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:54.148953 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9\": container with ID starting with 503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9 not found: ID does not exist" containerID="503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9" Nov 03 00:15:54.149044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.148980 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9"} err="failed to get container status \"503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9\": rpc error: code = NotFound desc = could not find container \"503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9\": container with ID starting with 503ad772264aee795a573deea77ba0dfba2ff0a0a7130a3a681dbddeb4048fc9 not found: ID does not exist" Nov 03 00:15:54.149044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.149002 5144 scope.go:122] "RemoveContainer" containerID="a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790" Nov 03 00:15:54.149210 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:15:54.149194 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790\": container with ID starting with a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790 not found: ID does not exist" containerID="a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790" Nov 03 00:15:54.149262 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.149219 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790"} err="failed to get container status \"a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790\": rpc error: code = NotFound desc = could not find container \"a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790\": container with ID starting with a8214821c3dac9a6ce366d23a05b8935a4790039e4c92f24e788a8442805c790 not found: ID does not exist" Nov 03 00:15:54.164342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.164314 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.164342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.164338 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgjsx\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-kube-api-access-kgjsx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.164453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.164353 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.164453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.164364 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6dc027a-bd51-43db-91a8-5cca7787a86b-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:54.315223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.315176 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74"] Nov 03 00:15:54.317621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.317601 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-12537192-openshift-default-546d597f7-h6h74"] Nov 03 00:15:54.426601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.426573 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24"] Nov 03 00:15:54.431277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:54.431252 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-01450ec4-openshift-default-6df774bbd9-drg24"] Nov 03 00:15:55.995052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:55.995018 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" path="/var/lib/kubelet/pods/72919027-3c7d-4b04-8fd1-5f92fcaf1b4f/volumes" Nov 03 00:15:55.995428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:55.995414 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" path="/var/lib/kubelet/pods/c6dc027a-bd51-43db-91a8-5cca7787a86b/volumes" Nov 03 00:15:58.263722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.263694 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw"] Nov 03 00:15:58.264140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264127 5144 cpu_manager.go:414] "containerMap: removing container" podUID="766cf582-ff64-4c0e-8276-30fcc749c761" containerName="revoke-keys" Nov 03 00:15:58.264184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264141 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766cf582-ff64-4c0e-8276-30fcc749c761" containerName="revoke-keys" Nov 03 00:15:58.264184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264165 5144 cpu_manager.go:414] "containerMap: removing container" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" Nov 03 00:15:58.264184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264170 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" Nov 03 00:15:58.264184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264179 5144 cpu_manager.go:414] "containerMap: removing container" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" Nov 03 00:15:58.264184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264185 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264191 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264196 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264209 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264213 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264275 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264282 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="72919027-3c7d-4b04-8fd1-5f92fcaf1b4f" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264287 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264294 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6dc027a-bd51-43db-91a8-5cca7787a86b" containerName="istio-proxy" Nov 03 00:15:58.264335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.264302 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="766cf582-ff64-4c0e-8276-30fcc749c761" containerName="revoke-keys" Nov 03 00:15:58.268999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.268985 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.323291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.323270 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw"] Nov 03 00:15:58.369422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.369393 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-048135-dockercfg-d9fz6\"" Nov 03 00:15:58.369652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.369462 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-048135\"" Nov 03 00:15:58.405968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.405934 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.406173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.406151 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pngh5\" (UniqueName: \"kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5\") pod \"payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.421619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.421581 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf"] Nov 03 00:15:58.425682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.425664 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.478753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.478730 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf"] Nov 03 00:15:58.506925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.506901 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.507052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.506973 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pngh5\" (UniqueName: \"kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5\") pod \"payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.507052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.507007 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.507052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.507044 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g279q\" (UniqueName: \"kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q\") pod \"payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.507982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.507957 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.579135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.579064 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pngh5\" (UniqueName: \"kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5\") pod \"payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.581600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.581575 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:58.608378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.608195 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.608378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.608268 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g279q\" (UniqueName: \"kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q\") pod \"payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.609061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.609038 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.665920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.665896 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g279q\" (UniqueName: \"kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q\") pod \"payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.738957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.738929 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:15:58.960808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:58.960776 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw"] Nov 03 00:15:58.962213 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:58.962183 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5510541_f988_47ff_8d94_4a1bbe2144cf.slice/crio-e83d959cb8d7baea504700766889be3e77016d69c7b17a63c4dedfbb56d687e8 WatchSource:0}: Error finding container e83d959cb8d7baea504700766889be3e77016d69c7b17a63c4dedfbb56d687e8: Status 404 returned error can't find the container with id e83d959cb8d7baea504700766889be3e77016d69c7b17a63c4dedfbb56d687e8 Nov 03 00:15:59.063219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.063181 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" podUID="7ed6fca2-3abe-4395-adba-a0cca9323361" containerName="payload-pre-processing" containerID="cri-o://ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102" gracePeriod=25 Nov 03 00:15:59.110978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.110957 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf"] Nov 03 00:15:59.122498 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:15:59.122464 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16fbaa05_5dbc_4b8f_a70a_41cbef2a13d6.slice/crio-b7de6533ac3216867ff3460e0b684d722e9a945cd291a9e7d34b46d8d15dc9e0 WatchSource:0}: Error finding container b7de6533ac3216867ff3460e0b684d722e9a945cd291a9e7d34b46d8d15dc9e0: Status 404 returned error can't find the container with id b7de6533ac3216867ff3460e0b684d722e9a945cd291a9e7d34b46d8d15dc9e0 Nov 03 00:15:59.147451 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.147424 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" event={"ID":"b5510541-f988-47ff-8d94-4a1bbe2144cf","Type":"ContainerStarted","Data":"f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0"} Nov 03 00:15:59.147550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.147479 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" event={"ID":"b5510541-f988-47ff-8d94-4a1bbe2144cf","Type":"ContainerStarted","Data":"e83d959cb8d7baea504700766889be3e77016d69c7b17a63c4dedfbb56d687e8"} Nov 03 00:15:59.147550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.147520 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:15:59.260211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.259671 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" podStartSLOduration=1.259650787 podStartE2EDuration="1.259650787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:59.259263742 +0000 UTC m=+649.404958219" watchObservedRunningTime="2026-11-03 00:15:59.259650787 +0000 UTC m=+649.405345246" Nov 03 00:15:59.309107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.309086 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:15:59.416145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.416119 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume\") pod \"7ed6fca2-3abe-4395-adba-a0cca9323361\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " Nov 03 00:15:59.416329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.416311 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4\" (UniqueName: \"kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4\") pod \"7ed6fca2-3abe-4395-adba-a0cca9323361\" (UID: \"7ed6fca2-3abe-4395-adba-a0cca9323361\") " Nov 03 00:15:59.416549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.416526 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume" pod "7ed6fca2-3abe-4395-adba-a0cca9323361" (UID: "7ed6fca2-3abe-4395-adba-a0cca9323361"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:15:59.416667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.416603 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ed6fca2-3abe-4395-adba-a0cca9323361-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:15:59.418445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.418422 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4" pod "7ed6fca2-3abe-4395-adba-a0cca9323361" (UID: "7ed6fca2-3abe-4395-adba-a0cca9323361"). InnerVolumeSpecName "kube-api-access-f6jf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:15:59.517247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:15:59.517223 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6jf4\" (UniqueName: \"kubernetes.io/projected/7ed6fca2-3abe-4395-adba-a0cca9323361-kube-api-access-f6jf4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:00.155146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.155111 5144 generic.go:356] "Generic (PLEG): container finished" podID="7ed6fca2-3abe-4395-adba-a0cca9323361" containerID="ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102" exitCode=0 Nov 03 00:16:00.155299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.155163 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" Nov 03 00:16:00.155299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.155197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" event={"ID":"7ed6fca2-3abe-4395-adba-a0cca9323361","Type":"ContainerDied","Data":"ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102"} Nov 03 00:16:00.155299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.155253 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz" event={"ID":"7ed6fca2-3abe-4395-adba-a0cca9323361","Type":"ContainerDied","Data":"842a3457fa51a80a0eaf471f75382cf63f006d17989daf8e74fbe47825e532f2"} Nov 03 00:16:00.155299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.155274 5144 scope.go:122] "RemoveContainer" containerID="ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102" Nov 03 00:16:00.157188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.157068 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" event={"ID":"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6","Type":"ContainerStarted","Data":"05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0"} Nov 03 00:16:00.157188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.157133 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" event={"ID":"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6","Type":"ContainerStarted","Data":"b7de6533ac3216867ff3460e0b684d722e9a945cd291a9e7d34b46d8d15dc9e0"} Nov 03 00:16:00.157374 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.157217 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:16:00.166242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.166223 5144 scope.go:122] "RemoveContainer" containerID="ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102" Nov 03 00:16:00.166548 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:00.166528 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102\": container with ID starting with ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102 not found: ID does not exist" containerID="ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102" Nov 03 00:16:00.166605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.166556 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102"} err="failed to get container status \"ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102\": rpc error: code = NotFound desc = could not find container \"ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102\": container with ID starting with ed8a3dcb01caa94bf466c5ef28f20690fc2d468401bf9f3e695f7bc708418102 not found: ID does not exist" Nov 03 00:16:00.325864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.325807 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" podStartSLOduration=2.325794447 podStartE2EDuration="2.325794447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:00.32489193 +0000 UTC m=+650.470586390" watchObservedRunningTime="2026-11-03 00:16:00.325794447 +0000 UTC m=+650.471488902" Nov 03 00:16:00.408832 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.408792 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2"] Nov 03 00:16:00.412073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.412045 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-168f474d-56kv2"] Nov 03 00:16:00.441497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.441473 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz"] Nov 03 00:16:00.444672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:00.444652 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-168f474d-74f7c5d6f5-6zscz"] Nov 03 00:16:01.875547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.875506 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8"] Nov 03 00:16:01.876226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.876207 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7ed6fca2-3abe-4395-adba-a0cca9323361" containerName="payload-pre-processing" Nov 03 00:16:01.876373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.876229 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ed6fca2-3abe-4395-adba-a0cca9323361" containerName="payload-pre-processing" Nov 03 00:16:01.876373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.876363 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ed6fca2-3abe-4395-adba-a0cca9323361" containerName="payload-pre-processing" Nov 03 00:16:01.881777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.881756 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942123 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942167 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942342 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ws9x\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942476 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942540 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942581 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942768 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942797 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.944712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.942814 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:01.948971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.947910 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8"] Nov 03 00:16:01.989920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:01.989890 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-e28e1a-openshift-default-dockercfg-4zwkf\"" Nov 03 00:16:02.004539 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.004512 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7352f42c-44ed-4ca9-a94c-25d8c7022d58" path="/var/lib/kubelet/pods/7352f42c-44ed-4ca9-a94c-25d8c7022d58/volumes" Nov 03 00:16:02.004890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.004876 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ed6fca2-3abe-4395-adba-a0cca9323361" path="/var/lib/kubelet/pods/7ed6fca2-3abe-4395-adba-a0cca9323361/volumes" Nov 03 00:16:02.043527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043492 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043531 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043587 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043684 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043715 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043737 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043770 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043793 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.043943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043857 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ws9x\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.044128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.043995 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.044288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.044234 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.044621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.044500 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.044725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.044538 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.044725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.044679 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.046168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.046144 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.046563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.046541 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.105659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.105605 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ws9x\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.106966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.106938 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.209787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.209713 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:02.722055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.722018 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8"] Nov 03 00:16:02.725937 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:02.725911 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e28b6f2_d0b2_4346_9a0b_e706d0ac3363.slice/crio-6fd31e87a8468d5bd96eddea59e543549a0e6db27617147bdfd97eabf79774db WatchSource:0}: Error finding container 6fd31e87a8468d5bd96eddea59e543549a0e6db27617147bdfd97eabf79774db: Status 404 returned error can't find the container with id 6fd31e87a8468d5bd96eddea59e543549a0e6db27617147bdfd97eabf79774db Nov 03 00:16:02.728354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.728316 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:02.728414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.728384 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:02.728414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:02.728411 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:03.190828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:03.190795 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" event={"ID":"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363","Type":"ContainerStarted","Data":"5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e"} Nov 03 00:16:03.191320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:03.190844 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" event={"ID":"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363","Type":"ContainerStarted","Data":"6fd31e87a8468d5bd96eddea59e543549a0e6db27617147bdfd97eabf79774db"} Nov 03 00:16:03.210249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:03.210220 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:03.215935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:03.215901 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:03.314054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:03.314000 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podStartSLOduration=2.313984144 podStartE2EDuration="2.313984144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:03.310809343 +0000 UTC m=+653.456503842" watchObservedRunningTime="2026-11-03 00:16:03.313984144 +0000 UTC m=+653.459678599" Nov 03 00:16:04.196763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:04.196731 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:04.197754 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:04.197732 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:05.841035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:05.841007 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8"] Nov 03 00:16:05.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:05.891151 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-2ccce380-nxms8"] Nov 03 00:16:05.935188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:05.935145 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws"] Nov 03 00:16:05.935462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:05.935430 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" containerID="cri-o://403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6" gracePeriod=30 Nov 03 00:16:05.995088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:05.995058 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f" path="/var/lib/kubelet/pods/8b0fb5b4-0d47-4076-b3ea-8e8d40b94d0f/volumes" Nov 03 00:16:06.503586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:06.503555 5144 patch_prober.go:29] interesting pod/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:16:06.503789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:06.503622 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:16:10.160611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:10.160584 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 00:16:11.172117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.172096 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 00:16:11.187557 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.187535 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:16:11.239895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.239834 5144 generic.go:356] "Generic (PLEG): container finished" podID="2a55c316-a586-4377-99eb-164c028773cf" containerID="403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6" exitCode=0 Nov 03 00:16:11.240011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.239896 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" Nov 03 00:16:11.240011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.239911 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" event={"ID":"2a55c316-a586-4377-99eb-164c028773cf","Type":"ContainerDied","Data":"403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6"} Nov 03 00:16:11.240011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.239963 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws" event={"ID":"2a55c316-a586-4377-99eb-164c028773cf","Type":"ContainerDied","Data":"2bada98383dd68f6510c9bb8b507c53d1ff63bcbd616c78ac17fd3db927f1558"} Nov 03 00:16:11.240011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.239990 5144 scope.go:122] "RemoveContainer" containerID="403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6" Nov 03 00:16:11.251177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.251160 5144 scope.go:122] "RemoveContainer" containerID="04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2" Nov 03 00:16:11.259947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.259924 5144 scope.go:122] "RemoveContainer" containerID="403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6" Nov 03 00:16:11.260240 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:11.260214 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6\": container with ID starting with 403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6 not found: ID does not exist" containerID="403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6" Nov 03 00:16:11.260307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.260248 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6"} err="failed to get container status \"403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6\": rpc error: code = NotFound desc = could not find container \"403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6\": container with ID starting with 403a87e8c19c28971df34072b3f3ce8f7e335151832fd852b242494b4009eff6 not found: ID does not exist" Nov 03 00:16:11.260307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.260267 5144 scope.go:122] "RemoveContainer" containerID="04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2" Nov 03 00:16:11.260541 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:11.260517 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2\": container with ID starting with 04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2 not found: ID does not exist" containerID="04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2" Nov 03 00:16:11.260594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.260551 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2"} err="failed to get container status \"04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2\": rpc error: code = NotFound desc = could not find container \"04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2\": container with ID starting with 04cd57cb12de4729706d8a8a3e4747717ff8d28fbf8bac7bd4957b05985f68b2 not found: ID does not exist" Nov 03 00:16:11.346381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.346342 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.346995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.346973 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.347304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.347287 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.347510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.347492 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.348114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.348094 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.348577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.348558 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.348957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.348940 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.349372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.349355 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.349872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.349854 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert\") pod \"2a55c316-a586-4377-99eb-164c028773cf\" (UID: \"2a55c316-a586-4377-99eb-164c028773cf\") " Nov 03 00:16:11.350786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.347393 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:11.350922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.348454 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:11.351021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.348879 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:11.351136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.349726 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:11.351221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.350268 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:11.351499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.351467 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "kube-api-access-ktv4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:11.353599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.353574 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:11.354980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.354953 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:11.355051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.354980 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo" pod "2a55c316-a586-4377-99eb-164c028773cf" (UID: "2a55c316-a586-4377-99eb-164c028773cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:16:11.451669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451592 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a55c316-a586-4377-99eb-164c028773cf-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451657 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktv4b\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-kube-api-access-ktv4b\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451672 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451684 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a55c316-a586-4377-99eb-164c028773cf-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451698 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a55c316-a586-4377-99eb-164c028773cf-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451711 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451725 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451739 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.451954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.451753 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a55c316-a586-4377-99eb-164c028773cf-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:11.662047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.662017 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws"] Nov 03 00:16:11.667840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.667807 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2ccce380-openshift-default-78fcfd6859-z7jws"] Nov 03 00:16:11.994795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:11.994728 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a55c316-a586-4377-99eb-164c028773cf" path="/var/lib/kubelet/pods/2a55c316-a586-4377-99eb-164c028773cf/volumes" Nov 03 00:16:12.155303 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:12.155268 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:16:13.219801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.219770 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2"] Nov 03 00:16:13.220275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.220226 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" Nov 03 00:16:13.220275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.220238 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" Nov 03 00:16:13.220275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.220250 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" Nov 03 00:16:13.220275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.220255 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" Nov 03 00:16:13.220464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.220335 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" Nov 03 00:16:13.220464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.220344 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a55c316-a586-4377-99eb-164c028773cf" containerName="istio-proxy" Nov 03 00:16:13.225078 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.225057 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.295670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.293966 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-e28e1a-dockercfg-rdd7r\"" Nov 03 00:16:13.295670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.294797 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-e28e1a\"" Nov 03 00:16:13.346485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.346446 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2"] Nov 03 00:16:13.348728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.348705 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm"] Nov 03 00:16:13.352396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.352381 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.387605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.387577 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qc8w\" (UniqueName: \"kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w\") pod \"payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.387792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.387722 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.399783 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.399756 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm"] Nov 03 00:16:13.489175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.489104 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.489306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.489183 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qc8w\" (UniqueName: \"kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w\") pod \"payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.489306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.489233 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlpnh\" (UniqueName: \"kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh\") pod \"payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.489306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.489251 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.489959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.489936 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.550068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.550048 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qc8w\" (UniqueName: \"kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w\") pod \"payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:13.590215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.590185 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.590336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.590304 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlpnh\" (UniqueName: \"kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh\") pod \"payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.590876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.590856 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.651755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.651729 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlpnh\" (UniqueName: \"kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh\") pod \"payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.665017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.664995 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:13.837789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:13.837753 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:14.048075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.048053 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm"] Nov 03 00:16:14.048824 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:14.048800 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803d1d10_ac58_4c88_98ad_27c52fe1de02.slice/crio-847d97f312269c9fb0f6f76f8e6ca317722d4f8940d6db22c18bec87277f2d5a WatchSource:0}: Error finding container 847d97f312269c9fb0f6f76f8e6ca317722d4f8940d6db22c18bec87277f2d5a: Status 404 returned error can't find the container with id 847d97f312269c9fb0f6f76f8e6ca317722d4f8940d6db22c18bec87277f2d5a Nov 03 00:16:14.214417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.214376 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2"] Nov 03 00:16:14.215459 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:14.215437 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod521293d7_4cb2_4ad3_b258_803887e4fbdf.slice/crio-634872c5ae4906d9ce1fffe3a39515a876051dd83af8f16d6c18dfa27f95099c WatchSource:0}: Error finding container 634872c5ae4906d9ce1fffe3a39515a876051dd83af8f16d6c18dfa27f95099c: Status 404 returned error can't find the container with id 634872c5ae4906d9ce1fffe3a39515a876051dd83af8f16d6c18dfa27f95099c Nov 03 00:16:14.260484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.260456 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" event={"ID":"521293d7-4cb2-4ad3-b258-803887e4fbdf","Type":"ContainerStarted","Data":"634872c5ae4906d9ce1fffe3a39515a876051dd83af8f16d6c18dfa27f95099c"} Nov 03 00:16:14.262221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.262196 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" event={"ID":"803d1d10-ac58-4c88-98ad-27c52fe1de02","Type":"ContainerStarted","Data":"c133cbc995a9bd4d968360e788b3904efc16fa5dc34f3067dbce02245d5beb5a"} Nov 03 00:16:14.262328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.262236 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" event={"ID":"803d1d10-ac58-4c88-98ad-27c52fe1de02","Type":"ContainerStarted","Data":"847d97f312269c9fb0f6f76f8e6ca317722d4f8940d6db22c18bec87277f2d5a"} Nov 03 00:16:14.262328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.262272 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:14.378868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:14.378776 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" podStartSLOduration=1.378762649 podStartE2EDuration="1.378762649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:14.376995616 +0000 UTC m=+664.522690087" watchObservedRunningTime="2026-11-03 00:16:14.378762649 +0000 UTC m=+664.524457139" Nov 03 00:16:15.269839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:15.269802 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" event={"ID":"521293d7-4cb2-4ad3-b258-803887e4fbdf","Type":"ContainerStarted","Data":"c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99"} Nov 03 00:16:15.383890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:15.383846 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" podStartSLOduration=2.383832471 podStartE2EDuration="2.383832471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:15.381885651 +0000 UTC m=+665.527580106" watchObservedRunningTime="2026-11-03 00:16:15.383832471 +0000 UTC m=+665.529526926" Nov 03 00:16:16.282663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:16.280364 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:22.917274 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:22.917205 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn"] Nov 03 00:16:22.920880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:22.920863 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:22.999674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:22.984753 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-531ee4\"" Nov 03 00:16:22.999674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:22.996156 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szg2f\" (UniqueName: \"kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f\") pod \"payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:22.999674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:22.996203 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:23.033720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.033687 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-531ee4-dockercfg-kdtwb\"" Nov 03 00:16:23.036104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.036083 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn"] Nov 03 00:16:23.085604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.085576 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn"] Nov 03 00:16:23.089836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.089815 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.096904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.096889 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szg2f\" (UniqueName: \"kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f\") pod \"payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:23.097017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.096923 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:23.097745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.097726 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:23.127190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.127167 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn"] Nov 03 00:16:23.153286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.153263 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szg2f\" (UniqueName: \"kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f\") pod \"payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:23.198154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.198105 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.198252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.198181 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztfpf\" (UniqueName: \"kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf\") pod \"payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.240014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.239996 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:23.300403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.300368 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.300546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.300527 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztfpf\" (UniqueName: \"kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf\") pod \"payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.301772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.301746 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.357666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.357616 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztfpf\" (UniqueName: \"kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf\") pod \"payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.403477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.403424 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:23.559989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.559955 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt"] Nov 03 00:16:23.567306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.567278 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-c6f27688-wvtjt"] Nov 03 00:16:23.626490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.626456 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn"] Nov 03 00:16:23.651861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.651832 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd"] Nov 03 00:16:23.652147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.652084 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" containerID="cri-o://e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e" gracePeriod=30 Nov 03 00:16:23.653128 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:23.653007 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a2f16b_0107_4a03_94c4_1a5abe76b7a0.slice/crio-2cc48b6fec2048b2639b0e06df3d8697011c6a32a0ba7153e8a3c50846469544 WatchSource:0}: Error finding container 2cc48b6fec2048b2639b0e06df3d8697011c6a32a0ba7153e8a3c50846469544: Status 404 returned error can't find the container with id 2cc48b6fec2048b2639b0e06df3d8697011c6a32a0ba7153e8a3c50846469544 Nov 03 00:16:23.782700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.782679 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn"] Nov 03 00:16:23.785405 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:23.785375 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda759ad9_314a_43b7_ab7d_597df96470cc.slice/crio-f696a1cc671c6d07454e60a087554c939738740362fe907edb6433783206389a WatchSource:0}: Error finding container f696a1cc671c6d07454e60a087554c939738740362fe907edb6433783206389a: Status 404 returned error can't find the container with id f696a1cc671c6d07454e60a087554c939738740362fe907edb6433783206389a Nov 03 00:16:23.995879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:23.995808 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766cf582-ff64-4c0e-8276-30fcc749c761" path="/var/lib/kubelet/pods/766cf582-ff64-4c0e-8276-30fcc749c761/volumes" Nov 03 00:16:24.336232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.336194 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" event={"ID":"da759ad9-314a-43b7-ab7d-597df96470cc","Type":"ContainerStarted","Data":"d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db"} Nov 03 00:16:24.336419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.336261 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" event={"ID":"da759ad9-314a-43b7-ab7d-597df96470cc","Type":"ContainerStarted","Data":"f696a1cc671c6d07454e60a087554c939738740362fe907edb6433783206389a"} Nov 03 00:16:24.336419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.336307 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:24.337752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.337724 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" event={"ID":"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0","Type":"ContainerStarted","Data":"0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9"} Nov 03 00:16:24.337890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.337775 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:24.337890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.337799 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" event={"ID":"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0","Type":"ContainerStarted","Data":"2cc48b6fec2048b2639b0e06df3d8697011c6a32a0ba7153e8a3c50846469544"} Nov 03 00:16:24.454645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.454577 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" podStartSLOduration=1.454560257 podStartE2EDuration="1.454560257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:24.451963103 +0000 UTC m=+674.597657581" watchObservedRunningTime="2026-11-03 00:16:24.454560257 +0000 UTC m=+674.600254712" Nov 03 00:16:24.569113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:24.569057 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" podStartSLOduration=2.569041405 podStartE2EDuration="2.569041405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:24.566672953 +0000 UTC m=+674.712367410" watchObservedRunningTime="2026-11-03 00:16:24.569041405 +0000 UTC m=+674.714735860" Nov 03 00:16:25.273190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:25.273164 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 00:16:25.982092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:25.982057 5144 patch_prober.go:29] interesting pod/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:16:25.982257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:25.982111 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:16:27.295101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:27.295067 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 00:16:28.899884 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.899860 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:16:28.967608 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967557 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967606 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967657 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967706 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967746 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967813 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967841 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967867 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.967958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967896 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:28.967958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967952 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data\") pod \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\" (UID: \"3ae357a1-8b3d-4210-b7f3-0edb0b961106\") " Nov 03 00:16:28.968192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.967975 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:28.968245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968202 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:28.968245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968229 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:28.968245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968229 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:16:28.968454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968433 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:28.968567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968457 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:28.968567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968469 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:28.968567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968481 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:28.968567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.968492 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:28.969954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.969931 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:16:28.970055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.969984 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:16:28.970378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.970362 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:28.970378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:28.970369 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7" pod "3ae357a1-8b3d-4210-b7f3-0edb0b961106" (UID: "3ae357a1-8b3d-4210-b7f3-0edb0b961106"). InnerVolumeSpecName "kube-api-access-xrps7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:16:29.069376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.069340 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrps7\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-kube-api-access-xrps7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:29.069376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.069371 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:29.069525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.069384 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:29.069525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.069396 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ae357a1-8b3d-4210-b7f3-0edb0b961106-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:16:29.367883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.367850 5144 generic.go:356] "Generic (PLEG): container finished" podID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerID="e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e" exitCode=0 Nov 03 00:16:29.368063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.367905 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" Nov 03 00:16:29.368063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.367937 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" event={"ID":"3ae357a1-8b3d-4210-b7f3-0edb0b961106","Type":"ContainerDied","Data":"e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e"} Nov 03 00:16:29.368063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.367986 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd" event={"ID":"3ae357a1-8b3d-4210-b7f3-0edb0b961106","Type":"ContainerDied","Data":"00debc95664f00f26d176f4073a2a859c1b6ba6c5ff66e8514cd55a493039807"} Nov 03 00:16:29.368063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.368008 5144 scope.go:122] "RemoveContainer" containerID="e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e" Nov 03 00:16:29.377910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.377893 5144 scope.go:122] "RemoveContainer" containerID="eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd" Nov 03 00:16:29.386233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.386219 5144 scope.go:122] "RemoveContainer" containerID="e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e" Nov 03 00:16:29.386457 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:29.386438 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e\": container with ID starting with e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e not found: ID does not exist" containerID="e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e" Nov 03 00:16:29.386515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.386472 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e"} err="failed to get container status \"e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e\": rpc error: code = NotFound desc = could not find container \"e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e\": container with ID starting with e610200108561286e971aa34e27c8dd24462e6c4cc16cb90e6d7e98ac0730e4e not found: ID does not exist" Nov 03 00:16:29.386515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.386488 5144 scope.go:122] "RemoveContainer" containerID="eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd" Nov 03 00:16:29.386752 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:29.386734 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd\": container with ID starting with eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd not found: ID does not exist" containerID="eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd" Nov 03 00:16:29.386822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.386763 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd"} err="failed to get container status \"eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd\": rpc error: code = NotFound desc = could not find container \"eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd\": container with ID starting with eba17c47d69d091394f42247acdf179618bb0dcdb808aeaaebc290a3484cfcfd not found: ID does not exist" Nov 03 00:16:29.535680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.535650 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd"] Nov 03 00:16:29.585002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.584980 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-c6f27688-gw-openshift-default-54c5ccff4f-ftlcd"] Nov 03 00:16:29.995979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:29.995946 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" path="/var/lib/kubelet/pods/3ae357a1-8b3d-4210-b7f3-0edb0b961106/volumes" Nov 03 00:16:35.346455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:35.346423 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 00:16:36.352229 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:36.352201 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 00:16:37.341404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.341362 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt"] Nov 03 00:16:37.341854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.341841 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" Nov 03 00:16:37.341854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.341855 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" Nov 03 00:16:37.341966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.341937 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" Nov 03 00:16:37.346621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.346604 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.399977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.399951 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt"] Nov 03 00:16:37.447946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.447834 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-d7bfc2-dockercfg-77c68\"" Nov 03 00:16:37.447946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.447917 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-d7bfc2\"" Nov 03 00:16:37.488513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.488484 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7"] Nov 03 00:16:37.489202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.489186 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" Nov 03 00:16:37.489297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.489287 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" Nov 03 00:16:37.489448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.489440 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ae357a1-8b3d-4210-b7f3-0edb0b961106" containerName="istio-proxy" Nov 03 00:16:37.490243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.490222 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.490485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.490466 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m8q7\" (UniqueName: \"kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7\") pod \"payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.492751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.492733 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.548845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.548813 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7"] Nov 03 00:16:37.591270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.591236 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m8q7\" (UniqueName: \"kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7\") pod \"payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.591471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.591292 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj29p\" (UniqueName: \"kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p\") pod \"payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.591471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.591414 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.591564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.591482 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.592163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.592137 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.648736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.648701 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m8q7\" (UniqueName: \"kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7\") pod \"payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.659571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.659540 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:37.692673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.692606 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj29p\" (UniqueName: \"kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p\") pod \"payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.692821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.692710 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.694509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.694475 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.753927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.753898 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj29p\" (UniqueName: \"kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p\") pod \"payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:37.808743 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:37.808714 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:38.060034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.060005 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt"] Nov 03 00:16:38.061247 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:38.061222 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaabe548_30c0_4efa_974d_400e2d40d98f.slice/crio-261ebdea25c6d63c9a778756962c3c0e14271c1daae92ff14cda4883230c1f62 WatchSource:0}: Error finding container 261ebdea25c6d63c9a778756962c3c0e14271c1daae92ff14cda4883230c1f62: Status 404 returned error can't find the container with id 261ebdea25c6d63c9a778756962c3c0e14271c1daae92ff14cda4883230c1f62 Nov 03 00:16:38.209054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.209017 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7"] Nov 03 00:16:38.211876 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:16:38.211852 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a6d2b26_8aa7_4697_ba17_7c537dee13cc.slice/crio-58a8e2bd37c73aec9f572dcab6abee88214f536fe6e59520276eb5e15edf86ed WatchSource:0}: Error finding container 58a8e2bd37c73aec9f572dcab6abee88214f536fe6e59520276eb5e15edf86ed: Status 404 returned error can't find the container with id 58a8e2bd37c73aec9f572dcab6abee88214f536fe6e59520276eb5e15edf86ed Nov 03 00:16:38.478041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.477950 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" event={"ID":"aaabe548-30c0-4efa-974d-400e2d40d98f","Type":"ContainerStarted","Data":"c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4"} Nov 03 00:16:38.478041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.478016 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:38.478480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.478067 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" event={"ID":"aaabe548-30c0-4efa-974d-400e2d40d98f","Type":"ContainerStarted","Data":"261ebdea25c6d63c9a778756962c3c0e14271c1daae92ff14cda4883230c1f62"} Nov 03 00:16:38.480319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.479910 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" event={"ID":"0a6d2b26-8aa7-4697-ba17-7c537dee13cc","Type":"ContainerStarted","Data":"91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00"} Nov 03 00:16:38.480319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.479948 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" event={"ID":"0a6d2b26-8aa7-4697-ba17-7c537dee13cc","Type":"ContainerStarted","Data":"58a8e2bd37c73aec9f572dcab6abee88214f536fe6e59520276eb5e15edf86ed"} Nov 03 00:16:38.480319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.479967 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:38.595860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.595404 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" podStartSLOduration=1.595387229 podStartE2EDuration="1.595387229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:38.594822648 +0000 UTC m=+688.740517104" watchObservedRunningTime="2026-11-03 00:16:38.595387229 +0000 UTC m=+688.741081679" Nov 03 00:16:38.711695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:38.711580 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" podStartSLOduration=1.711555079 podStartE2EDuration="1.711555079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:38.707853106 +0000 UTC m=+688.853547572" watchObservedRunningTime="2026-11-03 00:16:38.711555079 +0000 UTC m=+688.857249585" Nov 03 00:16:42.156833 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:16:42.156791 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:16:42.508037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:42.508002 5144 generic.go:356] "Generic (PLEG): container finished" podID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerID="5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e" exitCode=0 Nov 03 00:16:42.508206 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:42.508078 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" event={"ID":"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363","Type":"ContainerDied","Data":"5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e"} Nov 03 00:16:42.508502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:42.508473 5144 scope.go:122] "RemoveContainer" containerID="5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e" Nov 03 00:16:42.509846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:42.509817 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:42.509952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:42.509920 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:42.510020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:42.509970 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:16:43.533656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:43.533154 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" event={"ID":"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363","Type":"ContainerStarted","Data":"90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1"} Nov 03 00:16:44.210078 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:44.210041 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:44.210352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:44.210332 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:16:44.210408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:44.210380 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:16:45.210812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:45.210778 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:16:45.211287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:45.210872 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:16:46.210919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:46.210879 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:16:46.211318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:46.210955 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:16:47.210497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:47.210447 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:16:47.210842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:47.210514 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:16:48.210971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:48.210933 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 00:16:48.211376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:48.211010 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 00:16:49.197578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:49.197543 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:49.214756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:49.214726 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:16:49.489874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:49.489782 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 00:16:49.489874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:49.489866 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 00:16:49.584007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:16:49.583943 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 00:17:12.155759 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:17:12.155723 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:17:14.336662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.336615 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw"] Nov 03 00:17:14.340201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.340179 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.444164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.444130 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"openshift-service-ca.crt\"" Nov 03 00:17:14.444164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.444130 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"simulator-w0-ebeeb9-kserve-self-signed-certs\"" Nov 03 00:17:14.444385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.444211 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"default-dockercfg-64htb\"" Nov 03 00:17:14.444611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.444583 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"kube-root-ca.crt\"" Nov 03 00:17:14.446649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.446617 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw"] Nov 03 00:17:14.455029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.455003 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mmtk\" (UniqueName: \"kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.455183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.455065 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.455252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.455178 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.455252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.455208 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.455334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.455306 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.455370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.455343 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.556820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.556789 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mmtk\" (UniqueName: \"kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.556987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.556831 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.556987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.556853 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.556987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.556872 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.556987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.556949 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.557137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.556995 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.557309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.557282 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.557383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.557343 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.557433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.557382 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.559339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.559318 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.559492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.559478 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.614305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.614239 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mmtk\" (UniqueName: \"kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk\") pod \"simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:14.649979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:14.649949 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:15.021191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:15.021159 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw"] Nov 03 00:17:15.022317 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:17:15.022295 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod067644a1_d5bd_4cb2_a515_f73083ad711e.slice/crio-da5f867ad61c29ad92764a5325ea59a1ad3e8a04c5916d28e00312c3e5855d6e WatchSource:0}: Error finding container da5f867ad61c29ad92764a5325ea59a1ad3e8a04c5916d28e00312c3e5855d6e: Status 404 returned error can't find the container with id da5f867ad61c29ad92764a5325ea59a1ad3e8a04c5916d28e00312c3e5855d6e Nov 03 00:17:15.740518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:15.740481 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" event={"ID":"067644a1-d5bd-4cb2-a515-f73083ad711e","Type":"ContainerStarted","Data":"677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e"} Nov 03 00:17:15.740518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:15.740522 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" event={"ID":"067644a1-d5bd-4cb2-a515-f73083ad711e","Type":"ContainerStarted","Data":"da5f867ad61c29ad92764a5325ea59a1ad3e8a04c5916d28e00312c3e5855d6e"} Nov 03 00:17:15.856076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:15.856025 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podStartSLOduration=1.8560102729999999 podStartE2EDuration="1.856010273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:15.854256522 +0000 UTC m=+725.999951001" watchObservedRunningTime="2026-11-03 00:17:15.856010273 +0000 UTC m=+726.001704725" Nov 03 00:17:16.650565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:16.650526 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:24.650588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:24.650554 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:24.663085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:24.663062 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:24.816186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:24.816159 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:17:31.758838 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.758803 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn"] Nov 03 00:17:31.762692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.762675 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.859472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.859437 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ffzj\" (UniqueName: \"kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.859715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.859495 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.859715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.859598 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.859715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.859659 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.859715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.859683 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.859934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.859877 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.864725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.864703 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"kube-root-ca.crt\"" Nov 03 00:17:31.864899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.864756 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"openshift-service-ca.crt\"" Nov 03 00:17:31.864899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.864872 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"simulator-w2-531ee4-kserve-self-signed-certs\"" Nov 03 00:17:31.865036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.864928 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"default-dockercfg-br2w9\"" Nov 03 00:17:31.867575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.867553 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn"] Nov 03 00:17:31.961247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961205 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.961433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961263 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.961433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961302 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.961433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961395 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.961602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961449 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ffzj\" (UniqueName: \"kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.961602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961514 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.961778 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.961748 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.962140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.962110 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.962238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.962211 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.963700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.963677 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:31.963951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:31.963935 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:32.018505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:32.018419 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ffzj\" (UniqueName: \"kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj\") pod \"simulator-w2-531ee4-kserve-698bd48d84-8lzbn\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:32.075726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:32.075699 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:32.468085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:32.468017 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn"] Nov 03 00:17:32.860010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:32.859973 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" event={"ID":"3be973bd-a2eb-4026-a290-2aa35e7e08af","Type":"ContainerStarted","Data":"d4162ce811e0a971af5edb42f87d85795db00327f878a72e66a945afd0ef7d3f"} Nov 03 00:17:32.860505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:32.860023 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" event={"ID":"3be973bd-a2eb-4026-a290-2aa35e7e08af","Type":"ContainerStarted","Data":"f786f2c7450a73946959080fad1f4dd5eadf9881131e39b70313b2966670d57f"} Nov 03 00:17:32.978401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:32.978347 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" podStartSLOduration=1.978334958 podStartE2EDuration="1.978334958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:32.975764074 +0000 UTC m=+743.121458530" watchObservedRunningTime="2026-11-03 00:17:32.978334958 +0000 UTC m=+743.124029413" Nov 03 00:17:33.076617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:33.076583 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:42.076540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:42.076506 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:42.089578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:42.089554 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:42.154770 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:17:42.154741 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:17:42.936620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:42.936595 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 00:17:44.765210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.765178 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z"] Nov 03 00:17:44.769292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.769270 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.871588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.871535 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"openshift-service-ca.crt\"" Nov 03 00:17:44.871588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.871569 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"default-dockercfg-xsf2g\"" Nov 03 00:17:44.871588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.871575 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"kube-root-ca.crt\"" Nov 03 00:17:44.871878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.871666 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-048135\"/\"premium-w4-048135-kserve-self-signed-certs\"" Nov 03 00:17:44.874289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.874268 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z"] Nov 03 00:17:44.894647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.894607 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.894846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.894826 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.894924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.894867 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.894998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.894982 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2btbx\" (UniqueName: \"kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.895040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.895029 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.895074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.895047 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.995605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.995562 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2btbx\" (UniqueName: \"kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.995869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.995673 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.995869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.995704 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.995869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.995741 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.995869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.995853 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.996096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.995882 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.996168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.996146 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.996231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.996161 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.996279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.996229 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.998218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.998199 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:44.998459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:44.998443 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:45.052513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:45.052440 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2btbx\" (UniqueName: \"kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx\") pod \"premium-w4-048135-kserve-6c67f9c5c9-8qs8z\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:45.081780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:45.081749 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:45.458789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:45.458758 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z"] Nov 03 00:17:45.460047 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:17:45.460025 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32861b2c_92a4_4763_83eb_9b8e0eb3e083.slice/crio-333e8f727df7339dde8dcf719e0065a49b1d72de1014c1eaaaf41f5de15ae80a WatchSource:0}: Error finding container 333e8f727df7339dde8dcf719e0065a49b1d72de1014c1eaaaf41f5de15ae80a: Status 404 returned error can't find the container with id 333e8f727df7339dde8dcf719e0065a49b1d72de1014c1eaaaf41f5de15ae80a Nov 03 00:17:45.945293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:45.945257 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" event={"ID":"32861b2c-92a4-4763-83eb-9b8e0eb3e083","Type":"ContainerStarted","Data":"143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199"} Nov 03 00:17:45.945817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:45.945306 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" event={"ID":"32861b2c-92a4-4763-83eb-9b8e0eb3e083","Type":"ContainerStarted","Data":"333e8f727df7339dde8dcf719e0065a49b1d72de1014c1eaaaf41f5de15ae80a"} Nov 03 00:17:46.065035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:46.064973 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podStartSLOduration=2.064959407 podStartE2EDuration="2.064959407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:46.062110716 +0000 UTC m=+756.207805173" watchObservedRunningTime="2026-11-03 00:17:46.064959407 +0000 UTC m=+756.210653862" Nov 03 00:17:46.082453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:46.082414 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:55.082086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:55.082044 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:55.094339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:55.094314 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:17:56.019075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:17:56.019046 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 00:18:12.156149 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:18:12.156107 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:18:42.158033 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:18:42.157999 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:19:02.070121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.070085 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r"] Nov 03 00:19:02.075215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.075181 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.115552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.115517 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.115752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.115559 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wbrp\" (UniqueName: \"kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.115817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.115745 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.130033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.130006 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r"] Nov 03 00:19:02.176133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.176106 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:02.176303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.176266 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:19:02.217396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.217367 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.217546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.217454 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.217546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.217478 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wbrp\" (UniqueName: \"kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.218140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.218071 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.219892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.219871 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.275412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.275374 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wbrp\" (UniqueName: \"kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp\") pod \"maas-api-revoke-sub-5a89a9d9-dcb7r\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.392891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.392822 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:02.766468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:02.766437 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r"] Nov 03 00:19:02.767404 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:02.767382 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ef5712_612b_47d9_857c_63fc93dbbfba.slice/crio-e5abc49d4c422f038fe4917b4d646712ca0540a3ccf4ef90e0eb2c7a8f6e2392 WatchSource:0}: Error finding container e5abc49d4c422f038fe4917b4d646712ca0540a3ccf4ef90e0eb2c7a8f6e2392: Status 404 returned error can't find the container with id e5abc49d4c422f038fe4917b4d646712ca0540a3ccf4ef90e0eb2c7a8f6e2392 Nov 03 00:19:03.481667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:03.481564 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" event={"ID":"93ef5712-612b-47d9-857c-63fc93dbbfba","Type":"ContainerStarted","Data":"99355063a2f517f7dfd45cd8d159b9f864282a252fcf21d4f01deaac1316126e"} Nov 03 00:19:03.481667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:03.481626 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" event={"ID":"93ef5712-612b-47d9-857c-63fc93dbbfba","Type":"ContainerStarted","Data":"e5abc49d4c422f038fe4917b4d646712ca0540a3ccf4ef90e0eb2c7a8f6e2392"} Nov 03 00:19:03.595718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:03.595670 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" podStartSLOduration=1.595656232 podStartE2EDuration="1.595656232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:03.594672162 +0000 UTC m=+833.740366642" watchObservedRunningTime="2026-11-03 00:19:03.595656232 +0000 UTC m=+833.741350684" Nov 03 00:19:04.490830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:04.490800 5144 generic.go:356] "Generic (PLEG): container finished" podID="93ef5712-612b-47d9-857c-63fc93dbbfba" containerID="99355063a2f517f7dfd45cd8d159b9f864282a252fcf21d4f01deaac1316126e" exitCode=0 Nov 03 00:19:04.491253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:04.490895 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" event={"ID":"93ef5712-612b-47d9-857c-63fc93dbbfba","Type":"ContainerDied","Data":"99355063a2f517f7dfd45cd8d159b9f864282a252fcf21d4f01deaac1316126e"} Nov 03 00:19:05.643679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.643657 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:05.755457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.755413 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca\") pod \"93ef5712-612b-47d9-857c-63fc93dbbfba\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " Nov 03 00:19:05.755693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.755672 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp\" (UniqueName: \"kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp\") pod \"93ef5712-612b-47d9-857c-63fc93dbbfba\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " Nov 03 00:19:05.755963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.755736 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca\") pod \"93ef5712-612b-47d9-857c-63fc93dbbfba\" (UID: \"93ef5712-612b-47d9-857c-63fc93dbbfba\") " Nov 03 00:19:05.755963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.755938 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca" pod "93ef5712-612b-47d9-857c-63fc93dbbfba" (UID: "93ef5712-612b-47d9-857c-63fc93dbbfba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:05.756130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.756099 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:05.758718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.758687 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca" pod "93ef5712-612b-47d9-857c-63fc93dbbfba" (UID: "93ef5712-612b-47d9-857c-63fc93dbbfba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:05.758960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.758902 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp" pod "93ef5712-612b-47d9-857c-63fc93dbbfba" (UID: "93ef5712-612b-47d9-857c-63fc93dbbfba"). InnerVolumeSpecName "kube-api-access-8wbrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:05.856887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.856853 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wbrp\" (UniqueName: \"kubernetes.io/projected/93ef5712-612b-47d9-857c-63fc93dbbfba-kube-api-access-8wbrp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:05.856887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:05.856878 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93ef5712-612b-47d9-857c-63fc93dbbfba-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:06.158964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.158923 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d"] Nov 03 00:19:06.159428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.159413 5144 cpu_manager.go:414] "containerMap: removing container" podUID="93ef5712-612b-47d9-857c-63fc93dbbfba" containerName="revoke-keys" Nov 03 00:19:06.159851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.159428 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ef5712-612b-47d9-857c-63fc93dbbfba" containerName="revoke-keys" Nov 03 00:19:06.159851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.159590 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ef5712-612b-47d9-857c-63fc93dbbfba" containerName="revoke-keys" Nov 03 00:19:06.163136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.163116 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.217712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.217681 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"kube-root-ca.crt\"" Nov 03 00:19:06.262177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.262146 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.262370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.262180 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv28c\" (UniqueName: \"kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.262370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.262218 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.262506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.262430 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.262506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.262482 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.262575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.262558 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.266593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.266491 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"default-dockercfg-mpmcw\"" Nov 03 00:19:06.266593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.266496 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"premium-w3-d7bfc2-kserve-self-signed-certs\"" Nov 03 00:19:06.266593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.266529 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-d7bfc2\"/\"openshift-service-ca.crt\"" Nov 03 00:19:06.269383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.269357 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d"] Nov 03 00:19:06.364065 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364021 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364072 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364108 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364183 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364207 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv28c\" (UniqueName: \"kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364259 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364418 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364655 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.364991 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.364929 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.366511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.366490 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.366906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.366885 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.421327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.421297 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv28c\" (UniqueName: \"kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c\") pod \"premium-w3-d7bfc2-kserve-64bd465479-b562d\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.474903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.474868 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:06.504896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.504856 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" event={"ID":"93ef5712-612b-47d9-857c-63fc93dbbfba","Type":"ContainerDied","Data":"e5abc49d4c422f038fe4917b4d646712ca0540a3ccf4ef90e0eb2c7a8f6e2392"} Nov 03 00:19:06.505031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.504915 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5abc49d4c422f038fe4917b4d646712ca0540a3ccf4ef90e0eb2c7a8f6e2392" Nov 03 00:19:06.505031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.504875 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r" Nov 03 00:19:06.851994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.851965 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d"] Nov 03 00:19:06.852949 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:06.852919 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a5a804f_2741_40ac_81e3_372308a66d5e.slice/crio-7369c9c5d8532cb22a5cfb1588addbd9e6fe8af59752d791c4daacb1d7949001 WatchSource:0}: Error finding container 7369c9c5d8532cb22a5cfb1588addbd9e6fe8af59752d791c4daacb1d7949001: Status 404 returned error can't find the container with id 7369c9c5d8532cb22a5cfb1588addbd9e6fe8af59752d791c4daacb1d7949001 Nov 03 00:19:06.857372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.857350 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2"] Nov 03 00:19:06.857530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:06.857510 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" containerID="cri-o://8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8" gracePeriod=30 Nov 03 00:19:07.515303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:07.515261 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" event={"ID":"5a5a804f-2741-40ac-81e3-372308a66d5e","Type":"ContainerStarted","Data":"c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34"} Nov 03 00:19:07.515484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:07.515314 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" event={"ID":"5a5a804f-2741-40ac-81e3-372308a66d5e","Type":"ContainerStarted","Data":"7369c9c5d8532cb22a5cfb1588addbd9e6fe8af59752d791c4daacb1d7949001"} Nov 03 00:19:07.630512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:07.630448 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" podStartSLOduration=1.6304331109999999 podStartE2EDuration="1.630433111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:07.628937147 +0000 UTC m=+837.774631613" watchObservedRunningTime="2026-11-03 00:19:07.630433111 +0000 UTC m=+837.776127566" Nov 03 00:19:08.475295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.475258 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:08.744413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.744334 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr"] Nov 03 00:19:08.749072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.749051 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.787893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.787871 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85sr\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.787899 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.787942 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.788083 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.788131 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.788164 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.788186 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788248 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.788229 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.788351 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.788326 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.801070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.801040 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-531ee4-openshift-default-dockercfg-pcjd8\"" Nov 03 00:19:08.853815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.853783 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr"] Nov 03 00:19:08.889717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889684 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.889891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.889891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889769 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.889891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889787 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.889891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889806 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.889891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889826 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889921 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889957 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85sr\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.889984 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.890130 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.890303 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.890384 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.890442 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.890716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.890699 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.892412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.892387 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.892763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.892746 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.946391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.946357 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:08.946550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:08.946533 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85sr\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:09.063639 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:09.063594 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:09.492381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:09.492357 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr"] Nov 03 00:19:09.495094 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:09.495071 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2174356_8e04_4e28_9d10_a6041459d108.slice/crio-3148211c2ec0bda98649b2815b39d0cf0c9519789653d691087966e6c43f628b WatchSource:0}: Error finding container 3148211c2ec0bda98649b2815b39d0cf0c9519789653d691087966e6c43f628b: Status 404 returned error can't find the container with id 3148211c2ec0bda98649b2815b39d0cf0c9519789653d691087966e6c43f628b Nov 03 00:19:09.497240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:09.497208 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:09.497348 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:09.497312 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:09.497413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:09.497355 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:19:09.528975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:09.528946 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" event={"ID":"a2174356-8e04-4e28-9d10-a6041459d108","Type":"ContainerStarted","Data":"3148211c2ec0bda98649b2815b39d0cf0c9519789653d691087966e6c43f628b"} Nov 03 00:19:10.048648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:10.048542 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:19:10.048648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:10.048610 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:19:10.552710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:10.552670 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" event={"ID":"a2174356-8e04-4e28-9d10-a6041459d108","Type":"ContainerStarted","Data":"0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a"} Nov 03 00:19:10.779106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:10.779049 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podStartSLOduration=2.779035642 podStartE2EDuration="2.779035642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:10.727766625 +0000 UTC m=+840.873461084" watchObservedRunningTime="2026-11-03 00:19:10.779035642 +0000 UTC m=+840.924730151" Nov 03 00:19:11.063979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:11.063944 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:11.065476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:11.065447 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:19:11.065605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:11.065500 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 00:19:12.065778 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.065746 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:19:12.066220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.065819 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 00:19:12.181563 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:19:12.181526 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:19:12.187615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.187593 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:19:12.249150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249117 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249190 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249218 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249269 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249321 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249353 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249407 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249441 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249517 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy\") pod \"281353eb-04c5-4896-bf1e-b6948829ac54\" (UID: \"281353eb-04c5-4896-bf1e-b6948829ac54\") " Nov 03 00:19:12.249683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249594 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:12.249683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249659 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:12.249772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249731 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:12.249818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.249806 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:12.250108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.250019 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.250108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.250046 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.250108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.250061 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.250108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.250075 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/281353eb-04c5-4896-bf1e-b6948829ac54-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.250108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.250185 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:12.252136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.252113 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "kube-api-access-xrm48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:12.252239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.252224 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:12.252328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.252307 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:19:12.252377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.252325 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo" pod "281353eb-04c5-4896-bf1e-b6948829ac54" (UID: "281353eb-04c5-4896-bf1e-b6948829ac54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:19:12.354865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.353403 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrm48\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-kube-api-access-xrm48\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.354865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.353441 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.354865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.353455 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/281353eb-04c5-4896-bf1e-b6948829ac54-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.354865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.353469 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/281353eb-04c5-4896-bf1e-b6948829ac54-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.354865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.353482 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/281353eb-04c5-4896-bf1e-b6948829ac54-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:12.568740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.568704 5144 generic.go:356] "Generic (PLEG): container finished" podID="281353eb-04c5-4896-bf1e-b6948829ac54" containerID="8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8" exitCode=0 Nov 03 00:19:12.568914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.568757 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" Nov 03 00:19:12.568914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.568786 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" event={"ID":"281353eb-04c5-4896-bf1e-b6948829ac54","Type":"ContainerDied","Data":"8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8"} Nov 03 00:19:12.568914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.568847 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2" event={"ID":"281353eb-04c5-4896-bf1e-b6948829ac54","Type":"ContainerDied","Data":"e4d30ea589977a9a5c4fc7e707bb9810cabab909dbce1c629b85203678a17a9c"} Nov 03 00:19:12.568914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.568878 5144 scope.go:122] "RemoveContainer" containerID="8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8" Nov 03 00:19:12.578978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.578955 5144 scope.go:122] "RemoveContainer" containerID="2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c" Nov 03 00:19:12.587545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.587526 5144 scope.go:122] "RemoveContainer" containerID="8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8" Nov 03 00:19:12.587857 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:19:12.587838 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8\": container with ID starting with 8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8 not found: ID does not exist" containerID="8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8" Nov 03 00:19:12.587926 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.587863 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8"} err="failed to get container status \"8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8\": rpc error: code = NotFound desc = could not find container \"8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8\": container with ID starting with 8e22f5ab3d72b6ebe0f3efd11a8c3d03af91527c612e41d5bc711a8702f527d8 not found: ID does not exist" Nov 03 00:19:12.587926 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.587879 5144 scope.go:122] "RemoveContainer" containerID="2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c" Nov 03 00:19:12.588100 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:19:12.588087 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c\": container with ID starting with 2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c not found: ID does not exist" containerID="2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c" Nov 03 00:19:12.588135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.588104 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c"} err="failed to get container status \"2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c\": rpc error: code = NotFound desc = could not find container \"2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c\": container with ID starting with 2794f3854bf7c9e0e2f3fbcc5d7291847055585c530cb011ee01743c8fb8a42c not found: ID does not exist" Nov 03 00:19:12.747768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.747738 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2"] Nov 03 00:19:12.750855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:12.750824 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-qffb2"] Nov 03 00:19:13.064839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:13.064805 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:19:13.064998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:13.064873 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 00:19:14.005245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:14.005202 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" path="/var/lib/kubelet/pods/281353eb-04c5-4896-bf1e-b6948829ac54/volumes" Nov 03 00:19:14.064001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:14.063968 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:19:14.064185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:14.064024 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 00:19:15.064989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:15.064954 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:19:15.065384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:15.065012 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 00:19:16.475989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:16.475951 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:16.497399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:16.497370 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:16.614486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:16.614448 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:19:16.923533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:16.923498 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:16.924200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:16.924172 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:16.925097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:16.925075 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:19:19.874182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.874100 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6"] Nov 03 00:19:19.874870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.874850 5144 cpu_manager.go:414] "containerMap: removing container" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" Nov 03 00:19:19.874983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.874872 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" Nov 03 00:19:19.875041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.874998 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" Nov 03 00:19:19.875041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.875023 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" Nov 03 00:19:19.880044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.880024 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:19.936099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.936071 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6"] Nov 03 00:19:19.958135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.958105 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:19.958291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.958176 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdg4q\" (UniqueName: \"kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:19.958291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.958278 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:19.979362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.979335 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:19.979499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:19.979469 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:19:20.058933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.058904 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdg4q\" (UniqueName: \"kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.059102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.058947 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.059171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.059098 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.059734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.059713 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.061556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.061535 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.122311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.122282 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdg4q\" (UniqueName: \"kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q\") pod \"maas-api-revoke-sub-8fc9ead0-6cdd6\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.192958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.192887 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:20.568086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.568061 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6"] Nov 03 00:19:20.569556 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:20.569522 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ad5e6a3_aa92_4dc5_9a27_0133833f93ad.slice/crio-ac41a446b332b4602bb6545d1c07e47fd4fa10a3d24ce2450ba88fa7c9094a80 WatchSource:0}: Error finding container ac41a446b332b4602bb6545d1c07e47fd4fa10a3d24ce2450ba88fa7c9094a80: Status 404 returned error can't find the container with id ac41a446b332b4602bb6545d1c07e47fd4fa10a3d24ce2450ba88fa7c9094a80 Nov 03 00:19:20.633506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:20.633474 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" event={"ID":"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad","Type":"ContainerStarted","Data":"ac41a446b332b4602bb6545d1c07e47fd4fa10a3d24ce2450ba88fa7c9094a80"} Nov 03 00:19:21.641517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:21.641429 5144 generic.go:356] "Generic (PLEG): container finished" podID="8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" containerID="ff00670133ad83e38a25295ee8de7928f71d64835a05f50e99db17ba66ab47a1" exitCode=0 Nov 03 00:19:21.641517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:21.641490 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" event={"ID":"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad","Type":"ContainerDied","Data":"ff00670133ad83e38a25295ee8de7928f71d64835a05f50e99db17ba66ab47a1"} Nov 03 00:19:22.788189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.788167 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:22.890211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.890185 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q\" (UniqueName: \"kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q\") pod \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " Nov 03 00:19:22.890380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.890230 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca\") pod \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " Nov 03 00:19:22.890443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.890427 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca\") pod \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\" (UID: \"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad\") " Nov 03 00:19:22.890817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.890793 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca" pod "8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" (UID: "8ad5e6a3-aa92-4dc5-9a27-0133833f93ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:22.892471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.892453 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca" pod "8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" (UID: "8ad5e6a3-aa92-4dc5-9a27-0133833f93ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:22.892575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.892557 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q" pod "8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" (UID: "8ad5e6a3-aa92-4dc5-9a27-0133833f93ad"). InnerVolumeSpecName "kube-api-access-fdg4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:22.992326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.992292 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdg4q\" (UniqueName: \"kubernetes.io/projected/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-kube-api-access-fdg4q\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:22.992326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.992320 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:22.992504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:22.992332 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:23.655087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:23.655058 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" Nov 03 00:19:23.655252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:23.655092 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6" event={"ID":"8ad5e6a3-aa92-4dc5-9a27-0133833f93ad","Type":"ContainerDied","Data":"ac41a446b332b4602bb6545d1c07e47fd4fa10a3d24ce2450ba88fa7c9094a80"} Nov 03 00:19:23.655252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:23.655140 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac41a446b332b4602bb6545d1c07e47fd4fa10a3d24ce2450ba88fa7c9094a80" Nov 03 00:19:39.438700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.438663 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg"] Nov 03 00:19:39.439213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.439199 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" containerName="revoke-keys" Nov 03 00:19:39.439272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.439214 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" containerName="revoke-keys" Nov 03 00:19:39.439272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.439234 5144 cpu_manager.go:414] "containerMap: removing container" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" Nov 03 00:19:39.439272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.439239 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="281353eb-04c5-4896-bf1e-b6948829ac54" containerName="istio-proxy" Nov 03 00:19:39.439404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.439326 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" containerName="revoke-keys" Nov 03 00:19:39.443011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.442993 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.479085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.479055 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.479085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.479090 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.479300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.479208 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb86f\" (UniqueName: \"kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.495298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.495159 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:39.548023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.545840 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:19:39.549320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.549294 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg"] Nov 03 00:19:39.579887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.579848 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb86f\" (UniqueName: \"kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.580030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.579993 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.580030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.580015 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.580587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.580567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.582491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.582470 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.636343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.636316 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb86f\" (UniqueName: \"kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f\") pod \"maas-api-revoke-sub-6d806472-5pftg\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.756881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.756836 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:39.885022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.884987 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs"] Nov 03 00:19:39.889198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.889179 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:39.945658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.944649 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs"] Nov 03 00:19:39.984809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.984777 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4jq\" (UniqueName: \"kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:39.984980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.984818 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:39.984980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:39.984913 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.085802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.085719 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.085802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.085768 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.086008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.085955 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4jq\" (UniqueName: \"kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.086428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.086404 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.088283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.088261 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.145022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.144995 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg"] Nov 03 00:19:40.145190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.145161 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4jq\" (UniqueName: \"kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq\") pod \"maas-api-revoke-sub-3666f734-p6qbs\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.145498 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:40.145477 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc12acd0_fd1e_43c5_99b8_6c33bd71b4d5.slice/crio-4c6ad6f7930b8a28685f2b658968d0161beda60a3faad7d0bc0af19d7b93e250 WatchSource:0}: Error finding container 4c6ad6f7930b8a28685f2b658968d0161beda60a3faad7d0bc0af19d7b93e250: Status 404 returned error can't find the container with id 4c6ad6f7930b8a28685f2b658968d0161beda60a3faad7d0bc0af19d7b93e250 Nov 03 00:19:40.203150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.203131 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:40.580688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.580659 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs"] Nov 03 00:19:40.582068 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:40.582039 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3950b165_6af2_46e7_8a28_35d628d15297.slice/crio-e5e5f0358bd8e73f000eb686dbef35155fc5df4c92be7d464e0867a3ab0dac20 WatchSource:0}: Error finding container e5e5f0358bd8e73f000eb686dbef35155fc5df4c92be7d464e0867a3ab0dac20: Status 404 returned error can't find the container with id e5e5f0358bd8e73f000eb686dbef35155fc5df4c92be7d464e0867a3ab0dac20 Nov 03 00:19:40.769339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.769301 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" event={"ID":"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5","Type":"ContainerStarted","Data":"26a7af0a762426abce866d14600f0bf0a307386206383de678dda41c907ab55e"} Nov 03 00:19:40.769527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.769350 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" event={"ID":"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5","Type":"ContainerStarted","Data":"4c6ad6f7930b8a28685f2b658968d0161beda60a3faad7d0bc0af19d7b93e250"} Nov 03 00:19:40.770674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.770626 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" event={"ID":"3950b165-6af2-46e7-8a28-35d628d15297","Type":"ContainerStarted","Data":"e5e5f0358bd8e73f000eb686dbef35155fc5df4c92be7d464e0867a3ab0dac20"} Nov 03 00:19:40.881362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:40.881311 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" podStartSLOduration=1.8812957259999998 podStartE2EDuration="1.881295726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:40.880756853 +0000 UTC m=+871.026451308" watchObservedRunningTime="2026-11-03 00:19:40.881295726 +0000 UTC m=+871.026990184" Nov 03 00:19:41.778817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:41.778785 5144 generic.go:356] "Generic (PLEG): container finished" podID="fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" containerID="26a7af0a762426abce866d14600f0bf0a307386206383de678dda41c907ab55e" exitCode=0 Nov 03 00:19:41.779271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:41.778866 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" event={"ID":"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5","Type":"ContainerDied","Data":"26a7af0a762426abce866d14600f0bf0a307386206383de678dda41c907ab55e"} Nov 03 00:19:41.780642 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:41.780608 5144 generic.go:356] "Generic (PLEG): container finished" podID="3950b165-6af2-46e7-8a28-35d628d15297" containerID="f95211971d2eed3a45642e3219eab2b3bdae5c30f5d1e76dcc60d36cf4a0b63d" exitCode=0 Nov 03 00:19:41.780772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:41.780669 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" event={"ID":"3950b165-6af2-46e7-8a28-35d628d15297","Type":"ContainerDied","Data":"f95211971d2eed3a45642e3219eab2b3bdae5c30f5d1e76dcc60d36cf4a0b63d"} Nov 03 00:19:42.155255 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:19:42.155163 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:19:43.184706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.184677 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:43.190357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.190336 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:43.367161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367061 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca\") pod \"3950b165-6af2-46e7-8a28-35d628d15297\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " Nov 03 00:19:43.367161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367121 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f\" (UniqueName: \"kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f\") pod \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " Nov 03 00:19:43.367377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367182 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq\" (UniqueName: \"kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq\") pod \"3950b165-6af2-46e7-8a28-35d628d15297\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " Nov 03 00:19:43.367377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367214 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca\") pod \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " Nov 03 00:19:43.367377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367249 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca\") pod \"3950b165-6af2-46e7-8a28-35d628d15297\" (UID: \"3950b165-6af2-46e7-8a28-35d628d15297\") " Nov 03 00:19:43.367377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367323 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca\") pod \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\" (UID: \"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5\") " Nov 03 00:19:43.367841 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367807 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca" pod "3950b165-6af2-46e7-8a28-35d628d15297" (UID: "3950b165-6af2-46e7-8a28-35d628d15297"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:43.368065 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.367978 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca" pod "fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" (UID: "fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:43.369689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.369656 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca" pod "3950b165-6af2-46e7-8a28-35d628d15297" (UID: "3950b165-6af2-46e7-8a28-35d628d15297"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:43.369942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.369912 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq" pod "3950b165-6af2-46e7-8a28-35d628d15297" (UID: "3950b165-6af2-46e7-8a28-35d628d15297"). InnerVolumeSpecName "kube-api-access-wb4jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:43.370091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.370070 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f" pod "fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" (UID: "fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5"). InnerVolumeSpecName "kube-api-access-sb86f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:43.370180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.370162 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca" pod "fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" (UID: "fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:43.468494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.468455 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.468494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.468497 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb86f\" (UniqueName: \"kubernetes.io/projected/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-kube-api-access-sb86f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.468789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.468514 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4jq\" (UniqueName: \"kubernetes.io/projected/3950b165-6af2-46e7-8a28-35d628d15297-kube-api-access-wb4jq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.468789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.468530 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.468789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.468545 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3950b165-6af2-46e7-8a28-35d628d15297-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.468789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.468575 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:43.801981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.801944 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" event={"ID":"fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5","Type":"ContainerDied","Data":"4c6ad6f7930b8a28685f2b658968d0161beda60a3faad7d0bc0af19d7b93e250"} Nov 03 00:19:43.802170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.801998 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg" Nov 03 00:19:43.802170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.802030 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c6ad6f7930b8a28685f2b658968d0161beda60a3faad7d0bc0af19d7b93e250" Nov 03 00:19:43.803503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.803481 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" Nov 03 00:19:43.803646 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.803507 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs" event={"ID":"3950b165-6af2-46e7-8a28-35d628d15297","Type":"ContainerDied","Data":"e5e5f0358bd8e73f000eb686dbef35155fc5df4c92be7d464e0867a3ab0dac20"} Nov 03 00:19:43.803646 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:43.803555 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5e5f0358bd8e73f000eb686dbef35155fc5df4c92be7d464e0867a3ab0dac20" Nov 03 00:19:50.793031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.792992 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc"] Nov 03 00:19:50.793529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.793515 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3950b165-6af2-46e7-8a28-35d628d15297" containerName="revoke-keys" Nov 03 00:19:50.793574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.793531 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3950b165-6af2-46e7-8a28-35d628d15297" containerName="revoke-keys" Nov 03 00:19:50.793574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.793563 5144 cpu_manager.go:414] "containerMap: removing container" podUID="fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" containerName="revoke-keys" Nov 03 00:19:50.793574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.793568 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" containerName="revoke-keys" Nov 03 00:19:50.793699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.793687 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" containerName="revoke-keys" Nov 03 00:19:50.793735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.793708 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3950b165-6af2-46e7-8a28-35d628d15297" containerName="revoke-keys" Nov 03 00:19:50.798734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.798712 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.843750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.843718 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.843959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.843767 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.843959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.843803 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrmw\" (UniqueName: \"kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.855408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.855367 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc"] Nov 03 00:19:50.899471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.899437 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:50.899662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.899486 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:19:50.945186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.945147 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.945186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.945190 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrmw\" (UniqueName: \"kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.945456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.945285 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.945955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.945931 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:50.947728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:50.947706 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:51.001478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:51.001444 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrmw\" (UniqueName: \"kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw\") pod \"maas-api-revoke-sub-5ab4ae6c-tpkwc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:51.112770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:51.112685 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:51.491947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:51.491918 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc"] Nov 03 00:19:51.492870 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:51.492844 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda408be7a_3ba6_4e8f_b030_4cb874baf6fc.slice/crio-b9bf746a76a42d28a10ca7548fa114f86ce7bf84323e09712f96f7c8ebe7b2c5 WatchSource:0}: Error finding container b9bf746a76a42d28a10ca7548fa114f86ce7bf84323e09712f96f7c8ebe7b2c5: Status 404 returned error can't find the container with id b9bf746a76a42d28a10ca7548fa114f86ce7bf84323e09712f96f7c8ebe7b2c5 Nov 03 00:19:51.862029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:51.861992 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" event={"ID":"a408be7a-3ba6-4e8f-b030-4cb874baf6fc","Type":"ContainerStarted","Data":"07f9ca3a072c42420337c8b79eb26b0f94cf1e4aba749c72b2870cbd641bd51f"} Nov 03 00:19:51.862454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:51.862047 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" event={"ID":"a408be7a-3ba6-4e8f-b030-4cb874baf6fc","Type":"ContainerStarted","Data":"b9bf746a76a42d28a10ca7548fa114f86ce7bf84323e09712f96f7c8ebe7b2c5"} Nov 03 00:19:51.974218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:51.974162 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" podStartSLOduration=1.9741470890000001 podStartE2EDuration="1.974147089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:51.97372254 +0000 UTC m=+882.119417033" watchObservedRunningTime="2026-11-03 00:19:51.974147089 +0000 UTC m=+882.119841544" Nov 03 00:19:52.880570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:52.880534 5144 generic.go:356] "Generic (PLEG): container finished" podID="a408be7a-3ba6-4e8f-b030-4cb874baf6fc" containerID="07f9ca3a072c42420337c8b79eb26b0f94cf1e4aba749c72b2870cbd641bd51f" exitCode=0 Nov 03 00:19:52.881082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:52.880609 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" event={"ID":"a408be7a-3ba6-4e8f-b030-4cb874baf6fc","Type":"ContainerDied","Data":"07f9ca3a072c42420337c8b79eb26b0f94cf1e4aba749c72b2870cbd641bd51f"} Nov 03 00:19:54.064436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.064407 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:54.197061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.196976 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca\") pod \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " Nov 03 00:19:54.197061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.197031 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw\" (UniqueName: \"kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw\") pod \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " Nov 03 00:19:54.197296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.197065 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca\") pod \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\" (UID: \"a408be7a-3ba6-4e8f-b030-4cb874baf6fc\") " Nov 03 00:19:54.197431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.197401 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca" pod "a408be7a-3ba6-4e8f-b030-4cb874baf6fc" (UID: "a408be7a-3ba6-4e8f-b030-4cb874baf6fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:54.197581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.197561 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:54.199396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.199362 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw" pod "a408be7a-3ba6-4e8f-b030-4cb874baf6fc" (UID: "a408be7a-3ba6-4e8f-b030-4cb874baf6fc"). InnerVolumeSpecName "kube-api-access-wmrmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:54.199807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.199785 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca" pod "a408be7a-3ba6-4e8f-b030-4cb874baf6fc" (UID: "a408be7a-3ba6-4e8f-b030-4cb874baf6fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:54.299205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.299157 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrmw\" (UniqueName: \"kubernetes.io/projected/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-kube-api-access-wmrmw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:54.299205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.299190 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a408be7a-3ba6-4e8f-b030-4cb874baf6fc-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:54.905143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.905115 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" Nov 03 00:19:54.905327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.905160 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc" event={"ID":"a408be7a-3ba6-4e8f-b030-4cb874baf6fc","Type":"ContainerDied","Data":"b9bf746a76a42d28a10ca7548fa114f86ce7bf84323e09712f96f7c8ebe7b2c5"} Nov 03 00:19:54.905327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:54.905208 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9bf746a76a42d28a10ca7548fa114f86ce7bf84323e09712f96f7c8ebe7b2c5" Nov 03 00:19:55.097192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.097152 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s"] Nov 03 00:19:55.097689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.097674 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a408be7a-3ba6-4e8f-b030-4cb874baf6fc" containerName="revoke-keys" Nov 03 00:19:55.097755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.097690 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a408be7a-3ba6-4e8f-b030-4cb874baf6fc" containerName="revoke-keys" Nov 03 00:19:55.097821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.097812 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a408be7a-3ba6-4e8f-b030-4cb874baf6fc" containerName="revoke-keys" Nov 03 00:19:55.101255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.101239 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.154994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.154930 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:19:55.155296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.155219 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:19:55.159829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.159804 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s"] Nov 03 00:19:55.207940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.207897 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7tjb\" (UniqueName: \"kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.208138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.207948 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.208138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.208076 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.309712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.309676 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7tjb\" (UniqueName: \"kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.309928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.309729 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.309928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.309801 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.310722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.310579 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.312559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.312534 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.366455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.366403 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7tjb\" (UniqueName: \"kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb\") pod \"maas-api-revoke-sub-101e0521-q4v4s\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.414514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.414420 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:55.791150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.791119 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s"] Nov 03 00:19:55.792084 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:19:55.792037 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1f144c_5a70_499c_a84a_0e0f0bac5f66.slice/crio-33f369f8e95ac6eeb4f5108f26537449a94e150ea7d65069e1ab8017e5c2c03b WatchSource:0}: Error finding container 33f369f8e95ac6eeb4f5108f26537449a94e150ea7d65069e1ab8017e5c2c03b: Status 404 returned error can't find the container with id 33f369f8e95ac6eeb4f5108f26537449a94e150ea7d65069e1ab8017e5c2c03b Nov 03 00:19:55.913231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:55.913197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" event={"ID":"0a1f144c-5a70-499c-a84a-0e0f0bac5f66","Type":"ContainerStarted","Data":"33f369f8e95ac6eeb4f5108f26537449a94e150ea7d65069e1ab8017e5c2c03b"} Nov 03 00:19:56.922546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:56.922516 5144 generic.go:356] "Generic (PLEG): container finished" podID="0a1f144c-5a70-499c-a84a-0e0f0bac5f66" containerID="42409b4884ecf86b480232c44028295e5e6c86abd101bcce0ce82cea0246c2c7" exitCode=0 Nov 03 00:19:56.922971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:56.922575 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" event={"ID":"0a1f144c-5a70-499c-a84a-0e0f0bac5f66","Type":"ContainerDied","Data":"42409b4884ecf86b480232c44028295e5e6c86abd101bcce0ce82cea0246c2c7"} Nov 03 00:19:58.091488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.091460 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:58.241670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.241589 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca\") pod \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " Nov 03 00:19:58.241831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.241674 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca\") pod \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " Nov 03 00:19:58.241831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.241724 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb\" (UniqueName: \"kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb\") pod \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\" (UID: \"0a1f144c-5a70-499c-a84a-0e0f0bac5f66\") " Nov 03 00:19:58.241996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.241969 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca" pod "0a1f144c-5a70-499c-a84a-0e0f0bac5f66" (UID: "0a1f144c-5a70-499c-a84a-0e0f0bac5f66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:19:58.243912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.243890 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb" pod "0a1f144c-5a70-499c-a84a-0e0f0bac5f66" (UID: "0a1f144c-5a70-499c-a84a-0e0f0bac5f66"). InnerVolumeSpecName "kube-api-access-q7tjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:19:58.244029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.243968 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca" pod "0a1f144c-5a70-499c-a84a-0e0f0bac5f66" (UID: "0a1f144c-5a70-499c-a84a-0e0f0bac5f66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:19:58.343349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.343318 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7tjb\" (UniqueName: \"kubernetes.io/projected/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-kube-api-access-q7tjb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:58.343349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.343347 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:58.343349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.343356 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1f144c-5a70-499c-a84a-0e0f0bac5f66-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:19:58.940365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.940335 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" Nov 03 00:19:58.940548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.940325 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s" event={"ID":"0a1f144c-5a70-499c-a84a-0e0f0bac5f66","Type":"ContainerDied","Data":"33f369f8e95ac6eeb4f5108f26537449a94e150ea7d65069e1ab8017e5c2c03b"} Nov 03 00:19:58.940548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:19:58.940456 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33f369f8e95ac6eeb4f5108f26537449a94e150ea7d65069e1ab8017e5c2c03b" Nov 03 00:20:00.219789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.219750 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld"] Nov 03 00:20:00.220264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.220251 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0a1f144c-5a70-499c-a84a-0e0f0bac5f66" containerName="revoke-keys" Nov 03 00:20:00.220315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.220265 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1f144c-5a70-499c-a84a-0e0f0bac5f66" containerName="revoke-keys" Nov 03 00:20:00.220375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.220366 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1f144c-5a70-499c-a84a-0e0f0bac5f66" containerName="revoke-keys" Nov 03 00:20:00.223678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.223657 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.279761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.279726 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld"] Nov 03 00:20:00.325361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.325331 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:20:00.325540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.325379 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:20:00.363025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.362984 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.363025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.363027 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.363254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.363045 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwc5f\" (UniqueName: \"kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.463945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.463906 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.464153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.463952 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.464153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.463975 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwc5f\" (UniqueName: \"kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.464670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.464647 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.466507 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.466481 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.520406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.520367 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwc5f\" (UniqueName: \"kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f\") pod \"maas-api-revoke-sub-60cb67c3-gkzld\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.537286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.537251 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:00.913144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.913109 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld"] Nov 03 00:20:00.914476 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:20:00.914437 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd691217e_d678_44b0_a123_60e80a4dcfc3.slice/crio-d002fb26d597dd312409b9db6adc8834968e1060e2ff084bc160ae53f614058c WatchSource:0}: Error finding container d002fb26d597dd312409b9db6adc8834968e1060e2ff084bc160ae53f614058c: Status 404 returned error can't find the container with id d002fb26d597dd312409b9db6adc8834968e1060e2ff084bc160ae53f614058c Nov 03 00:20:00.957074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:00.957042 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" event={"ID":"d691217e-d678-44b0-a123-60e80a4dcfc3","Type":"ContainerStarted","Data":"d002fb26d597dd312409b9db6adc8834968e1060e2ff084bc160ae53f614058c"} Nov 03 00:20:01.967554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:01.967520 5144 generic.go:356] "Generic (PLEG): container finished" podID="d691217e-d678-44b0-a123-60e80a4dcfc3" containerID="8b5d816ef571533df0fa71541d37d287fd84ac22e2a36273849c9a2575d090f8" exitCode=0 Nov 03 00:20:01.968025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:01.967607 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" event={"ID":"d691217e-d678-44b0-a123-60e80a4dcfc3","Type":"ContainerDied","Data":"8b5d816ef571533df0fa71541d37d287fd84ac22e2a36273849c9a2575d090f8"} Nov 03 00:20:03.146023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.145994 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:03.305896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.305865 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca\") pod \"d691217e-d678-44b0-a123-60e80a4dcfc3\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " Nov 03 00:20:03.306079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.305987 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca\") pod \"d691217e-d678-44b0-a123-60e80a4dcfc3\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " Nov 03 00:20:03.306079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.306020 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f\" (UniqueName: \"kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f\") pod \"d691217e-d678-44b0-a123-60e80a4dcfc3\" (UID: \"d691217e-d678-44b0-a123-60e80a4dcfc3\") " Nov 03 00:20:03.306412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.306381 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca" pod "d691217e-d678-44b0-a123-60e80a4dcfc3" (UID: "d691217e-d678-44b0-a123-60e80a4dcfc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:03.308214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.308194 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f" pod "d691217e-d678-44b0-a123-60e80a4dcfc3" (UID: "d691217e-d678-44b0-a123-60e80a4dcfc3"). InnerVolumeSpecName "kube-api-access-jwc5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:03.308214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.308197 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca" pod "d691217e-d678-44b0-a123-60e80a4dcfc3" (UID: "d691217e-d678-44b0-a123-60e80a4dcfc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:03.407372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.407332 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:03.407372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.407373 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d691217e-d678-44b0-a123-60e80a4dcfc3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:03.407578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.407389 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwc5f\" (UniqueName: \"kubernetes.io/projected/d691217e-d678-44b0-a123-60e80a4dcfc3-kube-api-access-jwc5f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:03.983361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.983330 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" Nov 03 00:20:03.983541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.983366 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld" event={"ID":"d691217e-d678-44b0-a123-60e80a4dcfc3","Type":"ContainerDied","Data":"d002fb26d597dd312409b9db6adc8834968e1060e2ff084bc160ae53f614058c"} Nov 03 00:20:03.983541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:03.983434 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d002fb26d597dd312409b9db6adc8834968e1060e2ff084bc160ae53f614058c" Nov 03 00:20:10.245441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:10.245413 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:20:10.250964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:10.250941 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:20:11.006134 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.006099 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99"] Nov 03 00:20:11.006660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.006645 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d691217e-d678-44b0-a123-60e80a4dcfc3" containerName="revoke-keys" Nov 03 00:20:11.006730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.006660 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d691217e-d678-44b0-a123-60e80a4dcfc3" containerName="revoke-keys" Nov 03 00:20:11.006763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.006747 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d691217e-d678-44b0-a123-60e80a4dcfc3" containerName="revoke-keys" Nov 03 00:20:11.011396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.011375 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.066348 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.066317 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99"] Nov 03 00:20:11.111404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.111373 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:20:11.111571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.111431 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:20:11.188177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.188138 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.188177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.188179 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9vj\" (UniqueName: \"kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.188397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.188198 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.289944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.289850 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.289944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.289891 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9vj\" (UniqueName: \"kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.289944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.289910 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.290704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.290676 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.292382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.292363 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.347304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.347266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9vj\" (UniqueName: \"kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj\") pod \"maas-api-revoke-sub-0ecd699c-g2d99\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.628701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.626197 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:11.799364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.799328 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776"] Nov 03 00:20:11.804458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.804435 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.857882 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.857610 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7bfc2-openshift-default-dockercfg-g56sm\"" Nov 03 00:20:11.862877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.862841 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776"] Nov 03 00:20:11.999009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.998930 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.998986 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthgk\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999084 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999121 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999150 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999222 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999241 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999299 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:11.999415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:11.999373 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.004664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.004607 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99"] Nov 03 00:20:12.005319 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:20:12.005295 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed5b01a_b737_4afc_b2b2_d3d8ed008559.slice/crio-32a3048ecc0e87a867f184a5839c204e8f7090f408950ae7509a2f8dddb7ff85 WatchSource:0}: Error finding container 32a3048ecc0e87a867f184a5839c204e8f7090f408950ae7509a2f8dddb7ff85: Status 404 returned error can't find the container with id 32a3048ecc0e87a867f184a5839c204e8f7090f408950ae7509a2f8dddb7ff85 Nov 03 00:20:12.056640 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.056595 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" event={"ID":"fed5b01a-b737-4afc-b2b2-d3d8ed008559","Type":"ContainerStarted","Data":"32a3048ecc0e87a867f184a5839c204e8f7090f408950ae7509a2f8dddb7ff85"} Nov 03 00:20:12.100231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100203 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100263 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthgk\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100384 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100438 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100469 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100595 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100666 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100697 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100668 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100764 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.100860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.100798 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.101094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.101070 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.101203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.101095 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.101203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.101174 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.102930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.102902 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.103152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.103134 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.156652 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:20:12.156605 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:20:12.156789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.156720 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.156915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.156899 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthgk\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.419046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.418998 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:12.858127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.858092 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776"] Nov 03 00:20:12.859475 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:20:12.859449 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af72847_0d9a_4cc3_80b8_36689800df6d.slice/crio-fe20deb790cc3abeeac2f6a10066c334697c74372225bd894bdd13620f24e38f WatchSource:0}: Error finding container fe20deb790cc3abeeac2f6a10066c334697c74372225bd894bdd13620f24e38f: Status 404 returned error can't find the container with id fe20deb790cc3abeeac2f6a10066c334697c74372225bd894bdd13620f24e38f Nov 03 00:20:12.862541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.862509 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:12.862616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.862575 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:12.862616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:12.862607 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:13.064671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.064622 5144 generic.go:356] "Generic (PLEG): container finished" podID="fed5b01a-b737-4afc-b2b2-d3d8ed008559" containerID="ced1bb66d950cce96d6810baafbd740bbb5afb5725bd537964b1649d71b0f94d" exitCode=0 Nov 03 00:20:13.064836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.064671 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" event={"ID":"fed5b01a-b737-4afc-b2b2-d3d8ed008559","Type":"ContainerDied","Data":"ced1bb66d950cce96d6810baafbd740bbb5afb5725bd537964b1649d71b0f94d"} Nov 03 00:20:13.066370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.066339 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" event={"ID":"1af72847-0d9a-4cc3-80b8-36689800df6d","Type":"ContainerStarted","Data":"18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9"} Nov 03 00:20:13.066518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.066376 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" event={"ID":"1af72847-0d9a-4cc3-80b8-36689800df6d","Type":"ContainerStarted","Data":"fe20deb790cc3abeeac2f6a10066c334697c74372225bd894bdd13620f24e38f"} Nov 03 00:20:13.295712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.295610 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podStartSLOduration=2.295588618 podStartE2EDuration="2.295588618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:13.293247234 +0000 UTC m=+903.438941701" watchObservedRunningTime="2026-11-03 00:20:13.295588618 +0000 UTC m=+903.441283076" Nov 03 00:20:13.423004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.422189 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:13.426667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.426034 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:20:13.426667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.426122 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:20:13.621871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.621774 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5967f94-twfxc"] Nov 03 00:20:13.626993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.626968 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.682196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.682166 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5967f94-twfxc"] Nov 03 00:20:13.683020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.682994 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:20:13.683181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.683010 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:20:13.683274 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.683067 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a16401f7aab7\"" Nov 03 00:20:13.683336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.683109 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:20:13.683336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.683088 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0fb7528a8dea\"" Nov 03 00:20:13.683336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.683208 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:20:13.683336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.683234 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pnghk\"" Nov 03 00:20:13.749543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.749486 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.749914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.749887 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.750043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.749928 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.750113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.750041 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk7q\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.850751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.850714 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.850751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.850757 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.851013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.850783 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.851013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.850900 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk7q\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.851573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.851548 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.851703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.851598 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.853300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.853270 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.907644 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.907551 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk7q\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q\") pod \"payload-processing-external-model-ff5967f94-twfxc\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:13.954709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:13.954674 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:14.282049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.281797 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:14.357940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.357409 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5967f94-twfxc"] Nov 03 00:20:14.360433 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:20:14.360399 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9d0a9d_c069_4beb_8618_c22b0a4cee72.slice/crio-118f605c81778e0839ad75a11ea8dee27dc97a847a99bb7d89e26b6ea5a42198 WatchSource:0}: Error finding container 118f605c81778e0839ad75a11ea8dee27dc97a847a99bb7d89e26b6ea5a42198: Status 404 returned error can't find the container with id 118f605c81778e0839ad75a11ea8dee27dc97a847a99bb7d89e26b6ea5a42198 Nov 03 00:20:14.429669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.427592 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:20:14.429669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.427670 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:20:14.457831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.457164 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca\") pod \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " Nov 03 00:20:14.457831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.457293 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj\" (UniqueName: \"kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj\") pod \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " Nov 03 00:20:14.457831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.457326 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca\") pod \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\" (UID: \"fed5b01a-b737-4afc-b2b2-d3d8ed008559\") " Nov 03 00:20:14.458232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.458206 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca" pod "fed5b01a-b737-4afc-b2b2-d3d8ed008559" (UID: "fed5b01a-b737-4afc-b2b2-d3d8ed008559"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:20:14.463780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.462384 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca" pod "fed5b01a-b737-4afc-b2b2-d3d8ed008559" (UID: "fed5b01a-b737-4afc-b2b2-d3d8ed008559"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:20:14.463780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.462411 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj" pod "fed5b01a-b737-4afc-b2b2-d3d8ed008559" (UID: "fed5b01a-b737-4afc-b2b2-d3d8ed008559"). InnerVolumeSpecName "kube-api-access-ct9vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:20:14.558534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.558500 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9vj\" (UniqueName: \"kubernetes.io/projected/fed5b01a-b737-4afc-b2b2-d3d8ed008559-kube-api-access-ct9vj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:14.559075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.559054 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:14.559225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:14.559210 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fed5b01a-b737-4afc-b2b2-d3d8ed008559-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:20:15.086655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.085888 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" event={"ID":"1e9d0a9d-c069-4beb-8618-c22b0a4cee72","Type":"ContainerStarted","Data":"16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760"} Nov 03 00:20:15.086655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.085947 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" event={"ID":"1e9d0a9d-c069-4beb-8618-c22b0a4cee72","Type":"ContainerStarted","Data":"118f605c81778e0839ad75a11ea8dee27dc97a847a99bb7d89e26b6ea5a42198"} Nov 03 00:20:15.086655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.085997 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:15.087601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.087469 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" Nov 03 00:20:15.087601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.087509 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99" event={"ID":"fed5b01a-b737-4afc-b2b2-d3d8ed008559","Type":"ContainerDied","Data":"32a3048ecc0e87a867f184a5839c204e8f7090f408950ae7509a2f8dddb7ff85"} Nov 03 00:20:15.087601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.087545 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32a3048ecc0e87a867f184a5839c204e8f7090f408950ae7509a2f8dddb7ff85" Nov 03 00:20:15.202365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.202311 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" podStartSLOduration=2.202297977 podStartE2EDuration="2.202297977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:15.200572528 +0000 UTC m=+905.346267016" watchObservedRunningTime="2026-11-03 00:20:15.202297977 +0000 UTC m=+905.347992474" Nov 03 00:20:15.425940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.425849 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:20:15.426190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:15.426150 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:20:16.420657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:16.419954 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:20:16.420657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:16.420011 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:20:17.425658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:17.421078 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:20:17.425658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:17.421142 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:20:18.423141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:18.423100 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body= Nov 03 00:20:18.423438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:18.423409 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" Nov 03 00:20:19.468478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:19.468445 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:19.469348 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:19.469317 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:19.470341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:19.470322 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:20:22.448371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.448334 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq"] Nov 03 00:20:22.449003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.448983 5144 cpu_manager.go:414] "containerMap: removing container" podUID="fed5b01a-b737-4afc-b2b2-d3d8ed008559" containerName="revoke-keys" Nov 03 00:20:22.449003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.449003 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed5b01a-b737-4afc-b2b2-d3d8ed008559" containerName="revoke-keys" Nov 03 00:20:22.449147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.449109 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed5b01a-b737-4afc-b2b2-d3d8ed008559" containerName="revoke-keys" Nov 03 00:20:22.452498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.452481 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559544 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559581 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559604 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmp4f\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559802 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559824 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559853 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.559996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.559977 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.560077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.560053 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.560133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.560103 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.656444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.656410 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq"] Nov 03 00:20:22.661591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661560 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.661776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661602 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.661776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661671 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.661776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661727 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.661776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661754 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661778 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661873 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661904 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.661938 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmp4f\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.662266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.662269 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.662618 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.662896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.662873 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.663527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.663499 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.668933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.668910 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.669168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.669147 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.718986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.718923 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.719122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.719100 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmp4f\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:22.765993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:22.765968 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:23.198788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:23.198762 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq"] Nov 03 00:20:23.199554 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:20:23.199533 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd8e32b9_f553_406c_a3bb_38cb42456664.slice/crio-78b24f25566036ecdac05be1b3285bd3454b29c44757e93f4fefddfca6085a75 WatchSource:0}: Error finding container 78b24f25566036ecdac05be1b3285bd3454b29c44757e93f4fefddfca6085a75: Status 404 returned error can't find the container with id 78b24f25566036ecdac05be1b3285bd3454b29c44757e93f4fefddfca6085a75 Nov 03 00:20:23.201573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:23.201544 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:23.201646 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:23.201605 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:23.201691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:23.201661 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:24.151486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:24.151452 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" event={"ID":"fd8e32b9-f553-406c-a3bb-38cb42456664","Type":"ContainerStarted","Data":"d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0"} Nov 03 00:20:24.152040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:24.151502 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" event={"ID":"fd8e32b9-f553-406c-a3bb-38cb42456664","Type":"ContainerStarted","Data":"78b24f25566036ecdac05be1b3285bd3454b29c44757e93f4fefddfca6085a75"} Nov 03 00:20:24.335655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:24.320549 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podStartSLOduration=2.320530733 podStartE2EDuration="2.320530733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:24.319165469 +0000 UTC m=+914.464859954" watchObservedRunningTime="2026-11-03 00:20:24.320530733 +0000 UTC m=+914.466225190" Nov 03 00:20:24.766346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:24.766315 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:24.767964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:24.767938 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:20:24.768091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:24.767985 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 00:20:25.767185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:25.767155 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:20:25.767573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:25.767203 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 00:20:26.097445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:26.097375 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:20:26.767198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:26.767163 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:20:26.767569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:26.767225 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 00:20:27.767235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:27.767196 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:20:27.767625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:27.767257 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 00:20:28.767424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:28.767385 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:20:28.767859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:28.767462 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 00:20:29.766414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:29.766377 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 00:20:29.766590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:29.766458 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 00:20:31.485544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:31.485514 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:31.486007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:31.485985 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:31.486797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:31.486775 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:20:37.567878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.567843 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5"] Nov 03 00:20:37.573002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.572983 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663200 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663249 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663307 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663344 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663361 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663401 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwh57\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663587 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663759 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.663831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.663804 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764423 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764517 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764552 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764696 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764762 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764788 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764858 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764892 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.764992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.764953 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwh57\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.765163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.765073 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.765163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.765127 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.765305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.765281 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.765366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.765329 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.765572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.765548 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.767033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.767012 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.767447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.767424 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.775072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.775049 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5"] Nov 03 00:20:37.820621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.820564 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.820895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.820871 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwh57\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-995g5\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:37.886372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:37.886339 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:38.317302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:38.317275 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5"] Nov 03 00:20:38.318082 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:20:38.318057 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa60f82_9e28_423b_8788_0b28834b7052.slice/crio-0a4238f47706d2bd46a8883c5e6817cfa285bab90067e22a0b5e2ed16dd4adea WatchSource:0}: Error finding container 0a4238f47706d2bd46a8883c5e6817cfa285bab90067e22a0b5e2ed16dd4adea: Status 404 returned error can't find the container with id 0a4238f47706d2bd46a8883c5e6817cfa285bab90067e22a0b5e2ed16dd4adea Nov 03 00:20:38.320392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:38.320364 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.320468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:38.320426 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:38.320468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:38.320456 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:20:39.142567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.142529 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6"] Nov 03 00:20:39.147457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.147437 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.204097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.204072 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6"] Nov 03 00:20:39.255344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.255299 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" event={"ID":"eaa60f82-9e28-423b-8788-0b28834b7052","Type":"ContainerStarted","Data":"4ab4867f9633352ad9531c6805bb29d4443ec515cf54b4627a2e60528d9c3492"} Nov 03 00:20:39.255515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.255369 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" event={"ID":"eaa60f82-9e28-423b-8788-0b28834b7052","Type":"ContainerStarted","Data":"0a4238f47706d2bd46a8883c5e6817cfa285bab90067e22a0b5e2ed16dd4adea"} Nov 03 00:20:39.293399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.293367 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.293399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.293403 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.293583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.293518 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.293583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.293565 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgb96\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.374969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.374902 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podStartSLOduration=2.374885349 podStartE2EDuration="2.374885349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:39.374184835 +0000 UTC m=+929.519879290" watchObservedRunningTime="2026-11-03 00:20:39.374885349 +0000 UTC m=+929.520579806" Nov 03 00:20:39.395019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.394940 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.395019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.394979 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.395196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.395036 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.395196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.395151 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgb96\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.395784 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.395758 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.396076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.395786 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.397657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.397611 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.451315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.451282 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgb96\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96\") pod \"payload-processing-external-model-84fd5f4fc9-plst6\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.461201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.461178 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:39.863487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.859522 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6"] Nov 03 00:20:39.887150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.887131 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:39.888804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.888776 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:39.888892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:39.888829 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:40.264391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:40.264354 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" event={"ID":"ece90532-672d-4e1f-ba29-38dfc4755102","Type":"ContainerStarted","Data":"5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715"} Nov 03 00:20:40.264836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:40.264413 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" event={"ID":"ece90532-672d-4e1f-ba29-38dfc4755102","Type":"ContainerStarted","Data":"7f85fdbfbc00a24652668dd3a8d1ab46b836b9449818c692bacbac7ab5bc4fa7"} Nov 03 00:20:40.264836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:40.264527 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:20:40.392497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:40.392448 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" podStartSLOduration=1.3924336259999999 podStartE2EDuration="1.392433626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:40.391103886 +0000 UTC m=+930.536798363" watchObservedRunningTime="2026-11-03 00:20:40.392433626 +0000 UTC m=+930.538128083" Nov 03 00:20:40.887748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:40.887715 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:40.887916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:40.887774 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:41.887352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:41.887314 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:41.887739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:41.887386 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:42.174316 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:20:42.174225 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:20:42.886687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:42.886652 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:42.886902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:42.886721 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:43.887526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:43.887495 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:43.887893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:43.887548 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:44.886724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:44.886684 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:44.886905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:44.886751 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:45.887092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:45.887061 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:20:45.887530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:45.887507 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:20:47.887346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:47.887304 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:47.887904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:47.887402 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:48.928963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:48.928759 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:20:48.928963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:48.928815 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:20:49.250126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:49.250034 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:49.250883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:49.250855 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:49.251524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:49.251504 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:20:51.276676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:51.276619 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:20:59.164946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:59.164905 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6"] Nov 03 00:20:59.166146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:20:59.166123 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:21:04.165687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.165640 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" containerName="payload-processing" containerID="cri-o://5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715" gracePeriod=25 Nov 03 00:21:04.420713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.420661 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:21:04.511972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.511945 5144 generic.go:356] "Generic (PLEG): container finished" podID="ece90532-672d-4e1f-ba29-38dfc4755102" containerID="5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715" exitCode=0 Nov 03 00:21:04.512146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.511997 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" Nov 03 00:21:04.512146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.512024 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" event={"ID":"ece90532-672d-4e1f-ba29-38dfc4755102","Type":"ContainerDied","Data":"5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715"} Nov 03 00:21:04.512146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.512077 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6" event={"ID":"ece90532-672d-4e1f-ba29-38dfc4755102","Type":"ContainerDied","Data":"7f85fdbfbc00a24652668dd3a8d1ab46b836b9449818c692bacbac7ab5bc4fa7"} Nov 03 00:21:04.512146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.512099 5144 scope.go:122] "RemoveContainer" containerID="5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715" Nov 03 00:21:04.524354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.524319 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume\") pod \"ece90532-672d-4e1f-ba29-38dfc4755102\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " Nov 03 00:21:04.524507 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.524420 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials\") pod \"ece90532-672d-4e1f-ba29-38dfc4755102\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " Nov 03 00:21:04.524566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.524503 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96\") pod \"ece90532-672d-4e1f-ba29-38dfc4755102\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " Nov 03 00:21:04.524618 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.524583 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume\") pod \"ece90532-672d-4e1f-ba29-38dfc4755102\" (UID: \"ece90532-672d-4e1f-ba29-38dfc4755102\") " Nov 03 00:21:04.524771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.524698 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume" pod "ece90532-672d-4e1f-ba29-38dfc4755102" (UID: "ece90532-672d-4e1f-ba29-38dfc4755102"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:04.525113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.525081 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:04.525443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.525417 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume" pod "ece90532-672d-4e1f-ba29-38dfc4755102" (UID: "ece90532-672d-4e1f-ba29-38dfc4755102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:21:04.527048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.527029 5144 scope.go:122] "RemoveContainer" containerID="5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715" Nov 03 00:21:04.527416 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:04.527338 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715\": container with ID starting with 5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715 not found: ID does not exist" containerID="5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715" Nov 03 00:21:04.527416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.527392 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715"} err="failed to get container status \"5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715\": rpc error: code = NotFound desc = could not find container \"5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715\": container with ID starting with 5096c42157b0ecb3b01e6f87b365870af295abb7fa9098508e7dc70272d07715 not found: ID does not exist" Nov 03 00:21:04.527701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.527677 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96" pod "ece90532-672d-4e1f-ba29-38dfc4755102" (UID: "ece90532-672d-4e1f-ba29-38dfc4755102"). InnerVolumeSpecName "kube-api-access-sgb96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:04.530718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.530699 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials" pod "ece90532-672d-4e1f-ba29-38dfc4755102" (UID: "ece90532-672d-4e1f-ba29-38dfc4755102"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:21:04.625905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.625874 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgb96\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-kube-api-access-sgb96\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:04.625905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.625902 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ece90532-672d-4e1f-ba29-38dfc4755102-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:04.625905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.625912 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ece90532-672d-4e1f-ba29-38dfc4755102-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:21:04.941773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.941741 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6"] Nov 03 00:21:04.946610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:04.946582 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-84fd5f4fc9-plst6"] Nov 03 00:21:05.995553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:05.995520 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" path="/var/lib/kubelet/pods/ece90532-672d-4e1f-ba29-38dfc4755102/volumes" Nov 03 00:21:08.704956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.704921 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf"] Nov 03 00:21:08.705464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.705450 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" containerName="payload-processing" Nov 03 00:21:08.705509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.705465 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" containerName="payload-processing" Nov 03 00:21:08.705572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.705564 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ece90532-672d-4e1f-ba29-38dfc4755102" containerName="payload-processing" Nov 03 00:21:08.710291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.710266 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.765390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.765350 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf"] Nov 03 00:21:08.765908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.765882 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.766015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.765915 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.766077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.766014 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.766297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.766155 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.766297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.766183 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.766297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.766210 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhqk\" (UniqueName: \"kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.810767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.810737 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"distinct-w0-ebeeb9-kserve-self-signed-certs\"" Nov 03 00:21:08.867466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.867430 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.867739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.867476 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.867739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.867511 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhqk\" (UniqueName: \"kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.867739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.867650 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.867739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.867680 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.867989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.867738 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.868400 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.868376 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.871897 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.869465 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.871897 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.871712 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.874402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.874353 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.876305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.876284 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:08.926038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:08.926006 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhqk\" (UniqueName: \"kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk\") pod \"distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:09.025324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:09.025299 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:09.412364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:09.412329 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf"] Nov 03 00:21:09.413211 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:21:09.413187 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod474d0586_dcfe_478c_ae69_4baaaaeb0766.slice/crio-540270eb212bfd4ce159ea01442dfbe3651c96675b9c6923680d6be14abec908 WatchSource:0}: Error finding container 540270eb212bfd4ce159ea01442dfbe3651c96675b9c6923680d6be14abec908: Status 404 returned error can't find the container with id 540270eb212bfd4ce159ea01442dfbe3651c96675b9c6923680d6be14abec908 Nov 03 00:21:09.556175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:09.556136 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" event={"ID":"474d0586-dcfe-478c-ae69-4baaaaeb0766","Type":"ContainerStarted","Data":"92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6"} Nov 03 00:21:09.556344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:09.556188 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" event={"ID":"474d0586-dcfe-478c-ae69-4baaaaeb0766","Type":"ContainerStarted","Data":"540270eb212bfd4ce159ea01442dfbe3651c96675b9c6923680d6be14abec908"} Nov 03 00:21:09.671423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:09.671319 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" podStartSLOduration=1.671305386 podStartE2EDuration="1.671305386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:09.668463277 +0000 UTC m=+959.814157764" watchObservedRunningTime="2026-11-03 00:21:09.671305386 +0000 UTC m=+959.816999868" Nov 03 00:21:10.026285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:10.026254 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:12.154681 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:12.154613 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:21:19.026010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:19.025974 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:19.038512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:19.038437 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:19.636935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:19.636903 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:21:37.950270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:37.950236 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp"] Nov 03 00:21:37.954309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:37.954292 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003365 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003516 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003574 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003601 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003731 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64ff5\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003797 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.003905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003866 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.004013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003917 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.004013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.003953 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.008470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.008445 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-ebeeb9-openshift-default-dockercfg-pt5k5\"" Nov 03 00:21:38.061170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.061140 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp"] Nov 03 00:21:38.104927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.104873 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.105600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.105055 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.105761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.105360 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.105761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.105561 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.106018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.105997 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.106504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.106480 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.106661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.106585 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.106661 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:38.106620 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a16401f7aab7: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:38.106801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.106658 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.106801 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:38.106667 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5967f94-twfxc: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:38.106801 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:38.106726 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials podName:1e9d0a9d-c069-4beb-8618-c22b0a4cee72 nodeName:}" failed. No retries permitted until 2026-11-03 00:21:38.606707994 +0000 UTC m=+988.752402442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials") pod "payload-processing-external-model-ff5967f94-twfxc" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72") : secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:38.106801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.106765 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64ff5\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.107037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.106804 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.107037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.106886 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.107153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.107076 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.107447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.107426 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.107858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.107838 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.110104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.110079 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.110997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.110973 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.165486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.165442 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64ff5\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.167564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.167536 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.268260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.268221 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:38.618707 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:38.614619 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a16401f7aab7: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:38.618707 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:38.614672 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5967f94-twfxc: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:38.618707 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:38.614738 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials podName:1e9d0a9d-c069-4beb-8618-c22b0a4cee72 nodeName:}" failed. No retries permitted until 2026-11-03 00:21:39.61472016 +0000 UTC m=+989.760414607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials") pod "payload-processing-external-model-ff5967f94-twfxc" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72") : secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:38.716806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.715236 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp"] Nov 03 00:21:38.726362 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:21:38.726330 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacca85a_c92c_4051_84ad_a819ad5ff7ea.slice/crio-9c5c23ee0899610aa1909cb2b2edccfd4612c24d259425feed90a349d58f64c8 WatchSource:0}: Error finding container 9c5c23ee0899610aa1909cb2b2edccfd4612c24d259425feed90a349d58f64c8: Status 404 returned error can't find the container with id 9c5c23ee0899610aa1909cb2b2edccfd4612c24d259425feed90a349d58f64c8 Nov 03 00:21:38.732619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.731938 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:38.732619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.732039 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:38.732619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.732079 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:38.814858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:38.805321 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" event={"ID":"aacca85a-c92c-4051-84ad-a819ad5ff7ea","Type":"ContainerStarted","Data":"9c5c23ee0899610aa1909cb2b2edccfd4612c24d259425feed90a349d58f64c8"} Nov 03 00:21:39.639552 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:39.639510 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a16401f7aab7: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:39.639552 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:39.639550 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5967f94-twfxc: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:39.640120 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:39.639670 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials podName:1e9d0a9d-c069-4beb-8618-c22b0a4cee72 nodeName:}" failed. No retries permitted until 2026-11-03 00:21:41.639606144 +0000 UTC m=+991.785300583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials") pod "payload-processing-external-model-ff5967f94-twfxc" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72") : secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:39.813141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:39.813106 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" event={"ID":"aacca85a-c92c-4051-84ad-a819ad5ff7ea","Type":"ContainerStarted","Data":"7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c"} Nov 03 00:21:39.994677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:39.994547 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podStartSLOduration=2.994533977 podStartE2EDuration="2.994533977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:39.991808031 +0000 UTC m=+990.137502500" watchObservedRunningTime="2026-11-03 00:21:39.994533977 +0000 UTC m=+990.140228466" Nov 03 00:21:40.269275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:40.269236 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:40.270613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:40.270588 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:21:40.270787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:40.270646 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 00:21:41.268587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:41.268543 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:21:41.268990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:41.268606 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 00:21:41.667333 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:41.667252 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a16401f7aab7: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:41.667333 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:41.667277 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5967f94-twfxc: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:41.667333 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:41.667329 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials podName:1e9d0a9d-c069-4beb-8618-c22b0a4cee72 nodeName:}" failed. No retries permitted until 2026-11-03 00:21:45.667314662 +0000 UTC m=+995.813009096 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials") pod "payload-processing-external-model-ff5967f94-twfxc" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72") : secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:42.172296 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:42.172179 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:21:42.269075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:42.269045 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:21:42.269469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:42.269094 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 00:21:43.269439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:43.269400 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:21:43.269938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:43.269459 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 00:21:44.269170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:44.269132 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:21:44.269371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:44.269194 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 00:21:45.269563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:45.269530 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" start-of-body= Nov 03 00:21:45.270009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:45.269585 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.99:15021/healthz/ready\": dial tcp 10.132.0.99:15021: connect: connection refused" Nov 03 00:21:45.728230 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:45.728154 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a16401f7aab7: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:45.728230 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:45.728181 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5967f94-twfxc: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:45.728230 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:45.728230 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials podName:1e9d0a9d-c069-4beb-8618-c22b0a4cee72 nodeName:}" failed. No retries permitted until 2026-11-03 00:21:53.72821666 +0000 UTC m=+1003.873911094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials") pod "payload-processing-external-model-ff5967f94-twfxc" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72") : secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:46.273104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:46.273074 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:46.273551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:46.273437 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:46.273968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:46.273952 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:21:53.785644 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:53.785591 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a16401f7aab7: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:53.785644 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:53.785641 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-ff5967f94-twfxc: secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:53.786088 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:21:53.785702 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials podName:1e9d0a9d-c069-4beb-8618-c22b0a4cee72 nodeName:}" failed. No retries permitted until 2026-11-03 00:22:09.785685245 +0000 UTC m=+1019.931379681 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials") pod "payload-processing-external-model-ff5967f94-twfxc" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72") : secret "agc-cred-a16401f7aab7" not found Nov 03 00:21:57.615811 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.615779 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt"] Nov 03 00:21:57.620930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.620813 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.624800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.624771 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.624998 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625153 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625301 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625418 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78f4c\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625546 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625686 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625805 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.625943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.625931 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.719663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.719615 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt"] Nov 03 00:21:57.726995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.726957 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727049 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727086 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78f4c\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727139 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727186 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727214 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727268 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727334 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727381 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.727895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.727865 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.728110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.728089 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.728306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.728279 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.728982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.728907 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.729311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.729291 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.731808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.731782 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.731956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.731934 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.790826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.790788 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.791018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.790943 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78f4c\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:57.937178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:57.937098 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:58.133333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:58.133305 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5967f94-twfxc"] Nov 03 00:21:58.407080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:58.407050 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt"] Nov 03 00:21:58.408601 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:21:58.408572 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b630523_5039_4e27_b433_8033cb75d094.slice/crio-1d188a920dfafc36732e7ec5ca51f20f28f94b8eca85739b2f2deba76d1f1bd7 WatchSource:0}: Error finding container 1d188a920dfafc36732e7ec5ca51f20f28f94b8eca85739b2f2deba76d1f1bd7: Status 404 returned error can't find the container with id 1d188a920dfafc36732e7ec5ca51f20f28f94b8eca85739b2f2deba76d1f1bd7 Nov 03 00:21:58.411107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:58.411075 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:58.411194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:58.411160 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:58.411230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:58.411191 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:21:59.055095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:59.055055 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" event={"ID":"6b630523-5039-4e27-b433-8033cb75d094","Type":"ContainerStarted","Data":"17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273"} Nov 03 00:21:59.055579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:59.055108 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" event={"ID":"6b630523-5039-4e27-b433-8033cb75d094","Type":"ContainerStarted","Data":"1d188a920dfafc36732e7ec5ca51f20f28f94b8eca85739b2f2deba76d1f1bd7"} Nov 03 00:21:59.175101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:59.175051 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podStartSLOduration=2.175036699 podStartE2EDuration="2.175036699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:59.171016867 +0000 UTC m=+1009.316711344" watchObservedRunningTime="2026-11-03 00:21:59.175036699 +0000 UTC m=+1009.320731155" Nov 03 00:21:59.938724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:59.938689 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:21:59.939592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:59.939560 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:21:59.939740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:21:59.939623 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 00:22:00.938973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:00.938935 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:22:00.939561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:00.939529 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 00:22:01.943654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:01.938913 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:22:01.943654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:01.938968 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 00:22:02.939041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:02.939006 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:22:02.939258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:02.939066 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 00:22:03.137678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.137582 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" podUID="1e9d0a9d-c069-4beb-8618-c22b0a4cee72" containerName="payload-processing" containerID="cri-o://16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760" gracePeriod=25 Nov 03 00:22:03.464814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.464783 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:22:03.497906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.497876 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials\") pod \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " Nov 03 00:22:03.498069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.497988 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume\") pod \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " Nov 03 00:22:03.498069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.498018 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q\") pod \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " Nov 03 00:22:03.498069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.498035 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume\") pod \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\" (UID: \"1e9d0a9d-c069-4beb-8618-c22b0a4cee72\") " Nov 03 00:22:03.498479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.498450 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume" pod "1e9d0a9d-c069-4beb-8618-c22b0a4cee72" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:22:03.498479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.498459 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume" pod "1e9d0a9d-c069-4beb-8618-c22b0a4cee72" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:22:03.500155 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.500138 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials" pod "1e9d0a9d-c069-4beb-8618-c22b0a4cee72" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:22:03.500258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.500235 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q" pod "1e9d0a9d-c069-4beb-8618-c22b0a4cee72" (UID: "1e9d0a9d-c069-4beb-8618-c22b0a4cee72"). InnerVolumeSpecName "kube-api-access-hpk7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:22:03.599770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.599730 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:03.599770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.599768 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpk7q\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-kube-api-access-hpk7q\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:03.600019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.599783 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:03.600019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.599797 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1e9d0a9d-c069-4beb-8618-c22b0a4cee72-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:22:03.939015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.938926 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:22:03.939015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:03.938999 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 00:22:04.091872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.091841 5144 generic.go:356] "Generic (PLEG): container finished" podID="1e9d0a9d-c069-4beb-8618-c22b0a4cee72" containerID="16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760" exitCode=0 Nov 03 00:22:04.091872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.091872 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" Nov 03 00:22:04.092097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.091922 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" event={"ID":"1e9d0a9d-c069-4beb-8618-c22b0a4cee72","Type":"ContainerDied","Data":"16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760"} Nov 03 00:22:04.092097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.091972 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-ff5967f94-twfxc" event={"ID":"1e9d0a9d-c069-4beb-8618-c22b0a4cee72","Type":"ContainerDied","Data":"118f605c81778e0839ad75a11ea8dee27dc97a847a99bb7d89e26b6ea5a42198"} Nov 03 00:22:04.092097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.091994 5144 scope.go:122] "RemoveContainer" containerID="16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760" Nov 03 00:22:04.101941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.101926 5144 scope.go:122] "RemoveContainer" containerID="16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760" Nov 03 00:22:04.102199 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:22:04.102179 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760\": container with ID starting with 16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760 not found: ID does not exist" containerID="16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760" Nov 03 00:22:04.102252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.102201 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760"} err="failed to get container status \"16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760\": rpc error: code = NotFound desc = could not find container \"16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760\": container with ID starting with 16620f9c212c460d121c43ae3a4a1f4014b365e83d8b55dc42841c2f69437760 not found: ID does not exist" Nov 03 00:22:04.259489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.259456 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5967f94-twfxc"] Nov 03 00:22:04.309500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.309471 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-ff5967f94-twfxc"] Nov 03 00:22:04.937682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.937610 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body= Nov 03 00:22:04.937903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:04.937709 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" Nov 03 00:22:05.942191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:05.942160 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:22:05.942676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:05.942652 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:22:05.943430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:05.943413 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:22:05.996270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:05.996245 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e9d0a9d-c069-4beb-8618-c22b0a4cee72" path="/var/lib/kubelet/pods/1e9d0a9d-c069-4beb-8618-c22b0a4cee72/volumes" Nov 03 00:22:10.211656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:10.210872 5144 scope.go:122] "RemoveContainer" containerID="bfa099a276123f59e87dc9530977247541c157ca86ccecd559abe62bc57ed811" Nov 03 00:22:10.267654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:10.261766 5144 scope.go:122] "RemoveContainer" containerID="16af69988d7f80ee40ebadbf7f3f558f38a3318ee7534d51092714266ae83b87" Nov 03 00:22:10.308984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:10.308961 5144 scope.go:122] "RemoveContainer" containerID="2fdac7935ceb1aba0b1f16261d8efa54f537f78e7a2ee69924daae9034e97cab" Nov 03 00:22:10.384479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:10.384449 5144 scope.go:122] "RemoveContainer" containerID="5e99d9e05af101034032d3e5cf21d8bf23e00137d03635d94c743583a4da8551" Nov 03 00:22:10.400500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:10.400486 5144 scope.go:122] "RemoveContainer" containerID="fee665cc55f351c2dbed3955ca2430a200d8818649d46f746bd127d19ef33f00" Nov 03 00:22:12.156401 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:22:12.156357 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:22:42.157400 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:22:42.157359 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:22:52.916076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:52.916034 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk"] Nov 03 00:22:52.916831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:52.916809 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1e9d0a9d-c069-4beb-8618-c22b0a4cee72" containerName="payload-processing" Nov 03 00:22:52.916831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:52.916831 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e9d0a9d-c069-4beb-8618-c22b0a4cee72" containerName="payload-processing" Nov 03 00:22:52.917023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:52.917009 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e9d0a9d-c069-4beb-8618-c22b0a4cee72" containerName="payload-processing" Nov 03 00:22:52.920664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:52.920624 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:52.975705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:52.975675 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk"] Nov 03 00:22:53.021128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.021100 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-ebeeb9\"/\"distinct-2-w0-ebeeb9-kserve-self-signed-certs\"" Nov 03 00:22:53.023648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.023604 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nck4\" (UniqueName: \"kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.023780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.023678 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.023780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.023744 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.023861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.023827 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.023897 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.023870 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.023952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.023938 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.125539 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.125500 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.125769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.125546 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.125769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.125620 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.125908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.125804 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nck4\" (UniqueName: \"kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.125908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.125851 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.126015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.125910 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.126326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.126297 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.126483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.126338 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.126483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.126403 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.128669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.128646 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.128774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.128712 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.182418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.182327 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nck4\" (UniqueName: \"kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4\") pod \"distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.234581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.234548 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:22:53.625366 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:22:53.625335 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2513fc19_80f9_4ea6_b898_09c3a56164e0.slice/crio-6c9c03cf66f9b40276b13c21ebf1bfb1bd9d59f7131eccec8585887faeeac0c2 WatchSource:0}: Error finding container 6c9c03cf66f9b40276b13c21ebf1bfb1bd9d59f7131eccec8585887faeeac0c2: Status 404 returned error can't find the container with id 6c9c03cf66f9b40276b13c21ebf1bfb1bd9d59f7131eccec8585887faeeac0c2 Nov 03 00:22:53.625545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:53.625524 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk"] Nov 03 00:22:54.484650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:54.484606 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" event={"ID":"2513fc19-80f9-4ea6-b898-09c3a56164e0","Type":"ContainerStarted","Data":"834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02"} Nov 03 00:22:54.485040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:54.484670 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" event={"ID":"2513fc19-80f9-4ea6-b898-09c3a56164e0","Type":"ContainerStarted","Data":"6c9c03cf66f9b40276b13c21ebf1bfb1bd9d59f7131eccec8585887faeeac0c2"} Nov 03 00:22:54.600679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:54.600615 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podStartSLOduration=2.600600623 podStartE2EDuration="2.600600623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:54.598735906 +0000 UTC m=+1064.744430369" watchObservedRunningTime="2026-11-03 00:22:54.600600623 +0000 UTC m=+1064.746295078" Nov 03 00:22:55.235148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:22:55.235118 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:23:00.857673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:00.857598 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm"] Nov 03 00:23:00.862068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:00.862050 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:00.917950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:00.917931 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm"] Nov 03 00:23:00.962518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:00.962495 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:00.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:00.962573 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:23:01.024950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.024919 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.025151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.024972 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-626f7\" (UniqueName: \"kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.025151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.025075 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.125872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.125791 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.125996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.125985 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.126067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.126019 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-626f7\" (UniqueName: \"kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.127086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.127060 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.128339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.128319 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.187482 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.187449 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-626f7\" (UniqueName: \"kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7\") pod \"maas-api-revoke-sub-7fb3217d-8dncm\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.475443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.475372 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:01.850048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:01.850018 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm"] Nov 03 00:23:01.850452 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:01.850429 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b0a0d9_ddad_4dc3_a4a6_8698656f2bcb.slice/crio-2c5dad40529e95a4e8db028cdb6d839d1112abdea744162c7365d1a54544f275 WatchSource:0}: Error finding container 2c5dad40529e95a4e8db028cdb6d839d1112abdea744162c7365d1a54544f275: Status 404 returned error can't find the container with id 2c5dad40529e95a4e8db028cdb6d839d1112abdea744162c7365d1a54544f275 Nov 03 00:23:02.546548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:02.546514 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" event={"ID":"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb","Type":"ContainerStarted","Data":"5c17cae6217cf824870a8c034a32ab038613652012610f8f2fa3fb310b0b7ca3"} Nov 03 00:23:02.546992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:02.546573 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" event={"ID":"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb","Type":"ContainerStarted","Data":"2c5dad40529e95a4e8db028cdb6d839d1112abdea744162c7365d1a54544f275"} Nov 03 00:23:02.659530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:02.659469 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" podStartSLOduration=2.659455736 podStartE2EDuration="2.659455736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:02.658111835 +0000 UTC m=+1072.803806311" watchObservedRunningTime="2026-11-03 00:23:02.659455736 +0000 UTC m=+1072.805150220" Nov 03 00:23:03.235544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:03.235492 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:23:03.248488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:03.248461 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:23:03.557249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:03.557216 5144 generic.go:356] "Generic (PLEG): container finished" podID="42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" containerID="5c17cae6217cf824870a8c034a32ab038613652012610f8f2fa3fb310b0b7ca3" exitCode=0 Nov 03 00:23:03.557701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:03.557321 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" event={"ID":"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb","Type":"ContainerDied","Data":"5c17cae6217cf824870a8c034a32ab038613652012610f8f2fa3fb310b0b7ca3"} Nov 03 00:23:03.569092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:03.569059 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:23:04.753490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.753460 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:04.869112 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.869022 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca\") pod \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " Nov 03 00:23:04.869272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.869146 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7\" (UniqueName: \"kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7\") pod \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " Nov 03 00:23:04.869272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.869253 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca\") pod \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\" (UID: \"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb\") " Nov 03 00:23:04.869816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.869442 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca" pod "42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" (UID: "42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:04.869816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.869783 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:04.873023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.872995 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca" pod "42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" (UID: "42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:04.875014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.874989 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7" pod "42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" (UID: "42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb"). InnerVolumeSpecName "kube-api-access-626f7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:04.970742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.970712 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-626f7\" (UniqueName: \"kubernetes.io/projected/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-kube-api-access-626f7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:04.970742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:04.970738 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:05.574433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:05.574407 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" Nov 03 00:23:05.574595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:05.574428 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm" event={"ID":"42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb","Type":"ContainerDied","Data":"2c5dad40529e95a4e8db028cdb6d839d1112abdea744162c7365d1a54544f275"} Nov 03 00:23:05.574595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:05.574478 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5dad40529e95a4e8db028cdb6d839d1112abdea744162c7365d1a54544f275" Nov 03 00:23:12.158621 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:23:12.158581 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:23:16.860777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.860748 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7"] Nov 03 00:23:16.861505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.861485 5144 cpu_manager.go:414] "containerMap: removing container" podUID="42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" containerName="revoke-keys" Nov 03 00:23:16.861607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.861507 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" containerName="revoke-keys" Nov 03 00:23:16.861722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.861708 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" containerName="revoke-keys" Nov 03 00:23:16.865371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.865352 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:16.921008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.920985 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7"] Nov 03 00:23:16.966396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.966374 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:16.966534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:16.966456 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:23:17.029875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.029845 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwwjp\" (UniqueName: \"kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.029987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.029970 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.030045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.030028 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.131270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.131197 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.131380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.131278 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.131448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.131421 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwwjp\" (UniqueName: \"kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.131988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.131965 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.133706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.133688 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.187720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.187697 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwwjp\" (UniqueName: \"kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp\") pod \"maas-api-revoke-sub-ec751bd2-q6js7\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.479318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.479249 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:17.855045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:17.855014 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7"] Nov 03 00:23:17.855671 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:17.855649 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdf2d58_007e_485f_9634_00e72b118cdd.slice/crio-598fe8f8b49cfa8d4e820ec018c7f07fcc189b4b3b4d63857a59a5eeeabe1d29 WatchSource:0}: Error finding container 598fe8f8b49cfa8d4e820ec018c7f07fcc189b4b3b4d63857a59a5eeeabe1d29: Status 404 returned error can't find the container with id 598fe8f8b49cfa8d4e820ec018c7f07fcc189b4b3b4d63857a59a5eeeabe1d29 Nov 03 00:23:18.697676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:18.697641 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" event={"ID":"2cdf2d58-007e-485f-9634-00e72b118cdd","Type":"ContainerStarted","Data":"3322db2d1355865bbb36ac758f299ce7004b6007bd5c62a4cbcfe27afead9541"} Nov 03 00:23:18.698048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:18.697704 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" event={"ID":"2cdf2d58-007e-485f-9634-00e72b118cdd","Type":"ContainerStarted","Data":"598fe8f8b49cfa8d4e820ec018c7f07fcc189b4b3b4d63857a59a5eeeabe1d29"} Nov 03 00:23:18.812233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:18.812191 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" podStartSLOduration=2.8121792389999998 podStartE2EDuration="2.812179239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:18.809416342 +0000 UTC m=+1088.955110839" watchObservedRunningTime="2026-11-03 00:23:18.812179239 +0000 UTC m=+1088.957873695" Nov 03 00:23:19.708309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:19.708277 5144 generic.go:356] "Generic (PLEG): container finished" podID="2cdf2d58-007e-485f-9634-00e72b118cdd" containerID="3322db2d1355865bbb36ac758f299ce7004b6007bd5c62a4cbcfe27afead9541" exitCode=0 Nov 03 00:23:19.708789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:19.708362 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" event={"ID":"2cdf2d58-007e-485f-9634-00e72b118cdd","Type":"ContainerDied","Data":"3322db2d1355865bbb36ac758f299ce7004b6007bd5c62a4cbcfe27afead9541"} Nov 03 00:23:20.900923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:20.900892 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:21.071837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.071800 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca\") pod \"2cdf2d58-007e-485f-9634-00e72b118cdd\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " Nov 03 00:23:21.071837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.071845 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca\") pod \"2cdf2d58-007e-485f-9634-00e72b118cdd\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " Nov 03 00:23:21.072097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.071929 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp\" (UniqueName: \"kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp\") pod \"2cdf2d58-007e-485f-9634-00e72b118cdd\" (UID: \"2cdf2d58-007e-485f-9634-00e72b118cdd\") " Nov 03 00:23:21.072249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.072218 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca" pod "2cdf2d58-007e-485f-9634-00e72b118cdd" (UID: "2cdf2d58-007e-485f-9634-00e72b118cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:21.072417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.072401 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:21.074218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.074192 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp" pod "2cdf2d58-007e-485f-9634-00e72b118cdd" (UID: "2cdf2d58-007e-485f-9634-00e72b118cdd"). InnerVolumeSpecName "kube-api-access-bwwjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:21.074499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.074472 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca" pod "2cdf2d58-007e-485f-9634-00e72b118cdd" (UID: "2cdf2d58-007e-485f-9634-00e72b118cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:21.174040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.174016 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cdf2d58-007e-485f-9634-00e72b118cdd-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:21.174040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.174041 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwwjp\" (UniqueName: \"kubernetes.io/projected/2cdf2d58-007e-485f-9634-00e72b118cdd-kube-api-access-bwwjp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:21.725751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.725722 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" Nov 03 00:23:21.725919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.725763 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7" event={"ID":"2cdf2d58-007e-485f-9634-00e72b118cdd","Type":"ContainerDied","Data":"598fe8f8b49cfa8d4e820ec018c7f07fcc189b4b3b4d63857a59a5eeeabe1d29"} Nov 03 00:23:21.725919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:21.725817 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="598fe8f8b49cfa8d4e820ec018c7f07fcc189b4b3b4d63857a59a5eeeabe1d29" Nov 03 00:23:23.058546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.058516 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm"] Nov 03 00:23:23.059199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.059180 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2cdf2d58-007e-485f-9634-00e72b118cdd" containerName="revoke-keys" Nov 03 00:23:23.059262 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.059201 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdf2d58-007e-485f-9634-00e72b118cdd" containerName="revoke-keys" Nov 03 00:23:23.059334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.059324 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdf2d58-007e-485f-9634-00e72b118cdd" containerName="revoke-keys" Nov 03 00:23:23.063107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.063086 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.114947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.114924 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-048135-openshift-default-dockercfg-xt942\"" Nov 03 00:23:23.170299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.170274 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm"] Nov 03 00:23:23.199028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.198979 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvqxh\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199038 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199085 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199161 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199186 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199229 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199269 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199304 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.199521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.199419 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300541 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300576 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300742 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300784 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300822 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300956 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.300978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.300971 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301065 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvqxh\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301100 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301133 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301145 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301254 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301269 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.301729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.301710 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.302992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.302972 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.303564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.303548 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.357042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.356987 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.357129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.357118 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvqxh\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-f9ghm\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.378456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.378434 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:23.807550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.807518 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm"] Nov 03 00:23:23.807706 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:23.807681 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode131da3f_da7b_4026_99a9_e56a2789fb7a.slice/crio-67c6cd3a0dae3fe7b40040e1244b7caee15b62a83e3ba9b9ef197bb8c1408478 WatchSource:0}: Error finding container 67c6cd3a0dae3fe7b40040e1244b7caee15b62a83e3ba9b9ef197bb8c1408478: Status 404 returned error can't find the container with id 67c6cd3a0dae3fe7b40040e1244b7caee15b62a83e3ba9b9ef197bb8c1408478 Nov 03 00:23:23.809966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.809939 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:23.810040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.809998 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:23.810040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:23.810024 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:23:24.757923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:24.757885 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" event={"ID":"e131da3f-da7b-4026-99a9-e56a2789fb7a","Type":"ContainerStarted","Data":"eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330"} Nov 03 00:23:24.758497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:24.758475 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" event={"ID":"e131da3f-da7b-4026-99a9-e56a2789fb7a","Type":"ContainerStarted","Data":"67c6cd3a0dae3fe7b40040e1244b7caee15b62a83e3ba9b9ef197bb8c1408478"} Nov 03 00:23:25.379415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:25.379386 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:25.380886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:25.380865 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:23:25.380945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:25.380927 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 00:23:26.379171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:26.379137 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:23:26.379611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:26.379190 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 00:23:27.200846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.200787 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podStartSLOduration=4.200773143 podStartE2EDuration="4.200773143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:24.931870829 +0000 UTC m=+1095.077565319" watchObservedRunningTime="2026-11-03 00:23:27.200773143 +0000 UTC m=+1097.346467597" Nov 03 00:23:27.201476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.201455 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-l269t-q5pnz"] Nov 03 00:23:27.205389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.205369 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.263895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.263858 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l269t-q5pnz"] Nov 03 00:23:27.306686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.306651 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-klhlc\"" Nov 03 00:23:27.369468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.369436 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.369468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.369470 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55wch\" (UniqueName: \"kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.369707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.369495 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.369707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.369671 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.379409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.379387 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:23:27.379734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.379429 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 00:23:27.471046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.470954 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.471227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.471080 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.471227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.471100 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55wch\" (UniqueName: \"kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.471227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.471118 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.471483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.471460 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.471767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.471744 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.471851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.471821 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.546267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.546237 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55wch\" (UniqueName: \"kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch\") pod \"periodic-gathering-l269t-q5pnz\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:27.834882 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:27.834845 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:23:28.248163 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:28.248135 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d3dd78_bb5e_427c_808f_c25ad7e7a650.slice/crio-95a6e82028f9c1132516a6dcd7d750a8b7cf4540076bccd34a3b40d376189a93 WatchSource:0}: Error finding container 95a6e82028f9c1132516a6dcd7d750a8b7cf4540076bccd34a3b40d376189a93: Status 404 returned error can't find the container with id 95a6e82028f9c1132516a6dcd7d750a8b7cf4540076bccd34a3b40d376189a93 Nov 03 00:23:28.248366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:28.248342 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-l269t-q5pnz"] Nov 03 00:23:28.379137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:28.378954 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:23:28.379137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:28.379054 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 00:23:28.790703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:28.790661 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l269t-q5pnz" event={"ID":"69d3dd78-bb5e-427c-808f-c25ad7e7a650","Type":"ContainerStarted","Data":"b241070880efa4695742574e151849fb463027f6f5f5ebf1705bb4f4fbc9e98d"} Nov 03 00:23:28.791184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:28.790715 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l269t-q5pnz" event={"ID":"69d3dd78-bb5e-427c-808f-c25ad7e7a650","Type":"ContainerStarted","Data":"95a6e82028f9c1132516a6dcd7d750a8b7cf4540076bccd34a3b40d376189a93"} Nov 03 00:23:28.905955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:28.905902 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-l269t-q5pnz" podStartSLOduration=1.905887342 podStartE2EDuration="1.905887342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:28.901979115 +0000 UTC m=+1099.047673571" watchObservedRunningTime="2026-11-03 00:23:28.905887342 +0000 UTC m=+1099.051581824" Nov 03 00:23:29.379388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:29.379354 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:23:29.379574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:29.379405 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 00:23:30.379538 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:30.379490 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" start-of-body= Nov 03 00:23:30.379928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:30.379556 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": dial tcp 10.132.0.104:15021: connect: connection refused" Nov 03 00:23:32.380050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.380009 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:23:32.380529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.380067 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:23:32.447202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.447172 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:32.447689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.447666 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:32.448563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.448543 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:23:32.695765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.695678 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l"] Nov 03 00:23:32.699939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.699914 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.739185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.739151 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.739351 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.739311 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.739501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.739483 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkrvj\" (UniqueName: \"kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.760658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.758717 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:32.760658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.759088 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:23:32.765121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.765079 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l"] Nov 03 00:23:32.841020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.840989 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkrvj\" (UniqueName: \"kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.841178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.841082 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.841219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.841184 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.841985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.841963 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.843487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.843469 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:32.898063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:32.898037 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkrvj\" (UniqueName: \"kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj\") pod \"maas-api-revoke-sub-1112f7b8-wcb4l\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:33.023578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:33.023552 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:33.397576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:33.397548 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l"] Nov 03 00:23:33.398029 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:33.397598 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37aa31bc_faf8_4268_97ba_16114895693e.slice/crio-486f57589bb3cb9c6e8d1f9d3ddcbf1a1c3e76afb1acf92dccbb2ce1af68444d WatchSource:0}: Error finding container 486f57589bb3cb9c6e8d1f9d3ddcbf1a1c3e76afb1acf92dccbb2ce1af68444d: Status 404 returned error can't find the container with id 486f57589bb3cb9c6e8d1f9d3ddcbf1a1c3e76afb1acf92dccbb2ce1af68444d Nov 03 00:23:33.837718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:33.837684 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" event={"ID":"37aa31bc-faf8-4268-97ba-16114895693e","Type":"ContainerStarted","Data":"00ad5f52de7df46aaa87c515a71d34fa8bfa3fa8274403ec87a50f5411f20b8e"} Nov 03 00:23:33.837887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:33.837743 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" event={"ID":"37aa31bc-faf8-4268-97ba-16114895693e","Type":"ContainerStarted","Data":"486f57589bb3cb9c6e8d1f9d3ddcbf1a1c3e76afb1acf92dccbb2ce1af68444d"} Nov 03 00:23:33.953361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:33.953316 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" podStartSLOduration=1.9533028479999999 podStartE2EDuration="1.953302848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:33.949235646 +0000 UTC m=+1104.094930101" watchObservedRunningTime="2026-11-03 00:23:33.953302848 +0000 UTC m=+1104.098997303" Nov 03 00:23:34.672777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.672695 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4"] Nov 03 00:23:34.676929 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.676904 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.740653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.735846 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4"] Nov 03 00:23:34.765582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.765542 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.765774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.765693 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.765835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.765782 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj9l7\" (UniqueName: \"kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.846184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.846143 5144 generic.go:356] "Generic (PLEG): container finished" podID="37aa31bc-faf8-4268-97ba-16114895693e" containerID="00ad5f52de7df46aaa87c515a71d34fa8bfa3fa8274403ec87a50f5411f20b8e" exitCode=0 Nov 03 00:23:34.846184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.846189 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" event={"ID":"37aa31bc-faf8-4268-97ba-16114895693e","Type":"ContainerDied","Data":"00ad5f52de7df46aaa87c515a71d34fa8bfa3fa8274403ec87a50f5411f20b8e"} Nov 03 00:23:34.866728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.866703 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.866860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.866795 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.866906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.866860 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj9l7\" (UniqueName: \"kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.867494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.867425 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.869281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.869257 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.922944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.922886 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj9l7\" (UniqueName: \"kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7\") pod \"maas-api-revoke-sub-a7a19e6f-tnhj4\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:34.992323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:34.992296 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:35.373550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:35.373521 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4"] Nov 03 00:23:35.374109 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:35.374083 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ee6458a_af63_45f7_b542_78df123591ec.slice/crio-9b9a9d5faca4fd76c7a3303270e780f479f14e04aa4d826fd9451f21706165d3 WatchSource:0}: Error finding container 9b9a9d5faca4fd76c7a3303270e780f479f14e04aa4d826fd9451f21706165d3: Status 404 returned error can't find the container with id 9b9a9d5faca4fd76c7a3303270e780f479f14e04aa4d826fd9451f21706165d3 Nov 03 00:23:35.855347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:35.855313 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" event={"ID":"1ee6458a-af63-45f7-b542-78df123591ec","Type":"ContainerStarted","Data":"1d8b07a181bee37197b6955789c0155130575fd9c5f567d4dc9f06718a4711f7"} Nov 03 00:23:35.855751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:35.855374 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" event={"ID":"1ee6458a-af63-45f7-b542-78df123591ec","Type":"ContainerStarted","Data":"9b9a9d5faca4fd76c7a3303270e780f479f14e04aa4d826fd9451f21706165d3"} Nov 03 00:23:35.972186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:35.972091 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" podStartSLOduration=1.972072313 podStartE2EDuration="1.972072313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:35.969921625 +0000 UTC m=+1106.115616080" watchObservedRunningTime="2026-11-03 00:23:35.972072313 +0000 UTC m=+1106.117766770" Nov 03 00:23:35.998963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:35.998942 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:36.080873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.080849 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca\") pod \"37aa31bc-faf8-4268-97ba-16114895693e\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " Nov 03 00:23:36.080995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.080934 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca\") pod \"37aa31bc-faf8-4268-97ba-16114895693e\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " Nov 03 00:23:36.080995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.080961 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj\" (UniqueName: \"kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj\") pod \"37aa31bc-faf8-4268-97ba-16114895693e\" (UID: \"37aa31bc-faf8-4268-97ba-16114895693e\") " Nov 03 00:23:36.081297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.081272 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca" pod "37aa31bc-faf8-4268-97ba-16114895693e" (UID: "37aa31bc-faf8-4268-97ba-16114895693e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:36.083275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.083249 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj" pod "37aa31bc-faf8-4268-97ba-16114895693e" (UID: "37aa31bc-faf8-4268-97ba-16114895693e"). InnerVolumeSpecName "kube-api-access-rkrvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:36.083381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.083305 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca" pod "37aa31bc-faf8-4268-97ba-16114895693e" (UID: "37aa31bc-faf8-4268-97ba-16114895693e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:36.182384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.182357 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:36.182532 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.182388 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37aa31bc-faf8-4268-97ba-16114895693e-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:36.182532 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.182404 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkrvj\" (UniqueName: \"kubernetes.io/projected/37aa31bc-faf8-4268-97ba-16114895693e-kube-api-access-rkrvj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:36.865494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.865460 5144 generic.go:356] "Generic (PLEG): container finished" podID="1ee6458a-af63-45f7-b542-78df123591ec" containerID="1d8b07a181bee37197b6955789c0155130575fd9c5f567d4dc9f06718a4711f7" exitCode=0 Nov 03 00:23:36.865965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.865503 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" event={"ID":"1ee6458a-af63-45f7-b542-78df123591ec","Type":"ContainerDied","Data":"1d8b07a181bee37197b6955789c0155130575fd9c5f567d4dc9f06718a4711f7"} Nov 03 00:23:36.867047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.867028 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" Nov 03 00:23:36.867167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.867071 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l" event={"ID":"37aa31bc-faf8-4268-97ba-16114895693e","Type":"ContainerDied","Data":"486f57589bb3cb9c6e8d1f9d3ddcbf1a1c3e76afb1acf92dccbb2ce1af68444d"} Nov 03 00:23:36.867167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:36.867109 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="486f57589bb3cb9c6e8d1f9d3ddcbf1a1c3e76afb1acf92dccbb2ce1af68444d" Nov 03 00:23:38.031123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.031099 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:38.104560 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.104527 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7\" (UniqueName: \"kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7\") pod \"1ee6458a-af63-45f7-b542-78df123591ec\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " Nov 03 00:23:38.104751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.104593 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca\") pod \"1ee6458a-af63-45f7-b542-78df123591ec\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " Nov 03 00:23:38.104751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.104744 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca\") pod \"1ee6458a-af63-45f7-b542-78df123591ec\" (UID: \"1ee6458a-af63-45f7-b542-78df123591ec\") " Nov 03 00:23:38.104976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.104950 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca" pod "1ee6458a-af63-45f7-b542-78df123591ec" (UID: "1ee6458a-af63-45f7-b542-78df123591ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:38.105180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.105158 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:38.106830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.106807 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7" pod "1ee6458a-af63-45f7-b542-78df123591ec" (UID: "1ee6458a-af63-45f7-b542-78df123591ec"). InnerVolumeSpecName "kube-api-access-pj9l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:38.106932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.106830 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca" pod "1ee6458a-af63-45f7-b542-78df123591ec" (UID: "1ee6458a-af63-45f7-b542-78df123591ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:38.206318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.206252 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ee6458a-af63-45f7-b542-78df123591ec-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:38.206318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.206277 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj9l7\" (UniqueName: \"kubernetes.io/projected/1ee6458a-af63-45f7-b542-78df123591ec-kube-api-access-pj9l7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:38.882123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.882089 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" event={"ID":"1ee6458a-af63-45f7-b542-78df123591ec","Type":"ContainerDied","Data":"9b9a9d5faca4fd76c7a3303270e780f479f14e04aa4d826fd9451f21706165d3"} Nov 03 00:23:38.882296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.882126 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4" Nov 03 00:23:38.882296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:38.882142 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b9a9d5faca4fd76c7a3303270e780f479f14e04aa4d826fd9451f21706165d3" Nov 03 00:23:42.155712 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:23:42.155668 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:23:45.495196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.495166 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68"] Nov 03 00:23:45.495849 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.495833 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1ee6458a-af63-45f7-b542-78df123591ec" containerName="revoke-keys" Nov 03 00:23:45.495893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.495850 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ee6458a-af63-45f7-b542-78df123591ec" containerName="revoke-keys" Nov 03 00:23:45.495930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.495899 5144 cpu_manager.go:414] "containerMap: removing container" podUID="37aa31bc-faf8-4268-97ba-16114895693e" containerName="revoke-keys" Nov 03 00:23:45.495930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.495904 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37aa31bc-faf8-4268-97ba-16114895693e" containerName="revoke-keys" Nov 03 00:23:45.496010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.495986 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="37aa31bc-faf8-4268-97ba-16114895693e" containerName="revoke-keys" Nov 03 00:23:45.496010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.496001 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ee6458a-af63-45f7-b542-78df123591ec" containerName="revoke-keys" Nov 03 00:23:45.499713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.499694 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.557253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.557231 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:45.599023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.598993 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdkll\" (UniqueName: \"kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.599196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.599128 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.599196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.599168 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.606201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.606175 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:23:45.611665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.611624 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68"] Nov 03 00:23:45.700433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.700397 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdkll\" (UniqueName: \"kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.700585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.700514 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.700585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.700557 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.701335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.701313 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.703008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.702983 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.759568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.759545 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdkll\" (UniqueName: \"kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll\") pod \"maas-api-revoke-sub-dd859c9c-bdk68\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:45.813201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:45.813184 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:46.197595 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:23:46.197562 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0a72fc_f718_4e55_ab56_0f62f4bae131.slice/crio-b8acead96064c56d1c331fa45fe56b27b188d926ae869b1ab03eb694a0e93c6d WatchSource:0}: Error finding container b8acead96064c56d1c331fa45fe56b27b188d926ae869b1ab03eb694a0e93c6d: Status 404 returned error can't find the container with id b8acead96064c56d1c331fa45fe56b27b188d926ae869b1ab03eb694a0e93c6d Nov 03 00:23:46.198842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:46.198816 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68"] Nov 03 00:23:46.942469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:46.942386 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" event={"ID":"1e0a72fc-f718-4e55-ab56-0f62f4bae131","Type":"ContainerStarted","Data":"0e6e5aa13bb58639b0ebb43301cd6d9808735677320007bad17f06d97bb17a17"} Nov 03 00:23:46.942469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:46.942453 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" event={"ID":"1e0a72fc-f718-4e55-ab56-0f62f4bae131","Type":"ContainerStarted","Data":"b8acead96064c56d1c331fa45fe56b27b188d926ae869b1ab03eb694a0e93c6d"} Nov 03 00:23:47.062287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:47.062237 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" podStartSLOduration=2.062222262 podStartE2EDuration="2.062222262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:47.055725687 +0000 UTC m=+1117.201420175" watchObservedRunningTime="2026-11-03 00:23:47.062222262 +0000 UTC m=+1117.207916718" Nov 03 00:23:47.951761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:47.951722 5144 generic.go:356] "Generic (PLEG): container finished" podID="1e0a72fc-f718-4e55-ab56-0f62f4bae131" containerID="0e6e5aa13bb58639b0ebb43301cd6d9808735677320007bad17f06d97bb17a17" exitCode=0 Nov 03 00:23:47.952225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:47.951812 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" event={"ID":"1e0a72fc-f718-4e55-ab56-0f62f4bae131","Type":"ContainerDied","Data":"0e6e5aa13bb58639b0ebb43301cd6d9808735677320007bad17f06d97bb17a17"} Nov 03 00:23:48.675505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.675469 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql"] Nov 03 00:23:48.679366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.679351 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.736261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.736232 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql"] Nov 03 00:23:48.835864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.835801 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jzr\" (UniqueName: \"kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.835864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.835844 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.836045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.835962 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.937540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.937458 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jzr\" (UniqueName: \"kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.937540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.937521 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.937773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.937582 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.938144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.938122 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.940066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.940045 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:48.993813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:48.993780 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jzr\" (UniqueName: \"kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr\") pod \"maas-api-revoke-sub-0bdec8d5-khbql\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:49.113135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.113112 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:49.241154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.241083 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll\" (UniqueName: \"kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll\") pod \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " Nov 03 00:23:49.241154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.241114 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca\") pod \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " Nov 03 00:23:49.241154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.241135 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca\") pod \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\" (UID: \"1e0a72fc-f718-4e55-ab56-0f62f4bae131\") " Nov 03 00:23:49.241826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.241798 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca" pod "1e0a72fc-f718-4e55-ab56-0f62f4bae131" (UID: "1e0a72fc-f718-4e55-ab56-0f62f4bae131"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:49.243841 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.243810 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca" pod "1e0a72fc-f718-4e55-ab56-0f62f4bae131" (UID: "1e0a72fc-f718-4e55-ab56-0f62f4bae131"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:49.243961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.243942 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll" pod "1e0a72fc-f718-4e55-ab56-0f62f4bae131" (UID: "1e0a72fc-f718-4e55-ab56-0f62f4bae131"). InnerVolumeSpecName "kube-api-access-kdkll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:49.292714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.292691 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:49.343444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.343403 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdkll\" (UniqueName: \"kubernetes.io/projected/1e0a72fc-f718-4e55-ab56-0f62f4bae131-kube-api-access-kdkll\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:49.343444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.343443 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:49.343751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.343460 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0a72fc-f718-4e55-ab56-0f62f4bae131-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:49.668975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.668951 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql"] Nov 03 00:23:49.968625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.968599 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" event={"ID":"2a5e3786-63f4-4a67-910c-53320c5e1879","Type":"ContainerStarted","Data":"7cbc662af8aed8e9be7e6a1eaf97553ec9825fdc747f2ac5a34144030d09210b"} Nov 03 00:23:49.969911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.969886 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" event={"ID":"1e0a72fc-f718-4e55-ab56-0f62f4bae131","Type":"ContainerDied","Data":"b8acead96064c56d1c331fa45fe56b27b188d926ae869b1ab03eb694a0e93c6d"} Nov 03 00:23:49.969911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.969908 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68" Nov 03 00:23:49.970073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:49.969935 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8acead96064c56d1c331fa45fe56b27b188d926ae869b1ab03eb694a0e93c6d" Nov 03 00:23:50.977881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:50.977796 5144 generic.go:356] "Generic (PLEG): container finished" podID="2a5e3786-63f4-4a67-910c-53320c5e1879" containerID="f9a3a5d8073634c4711f223bfbe54622fc778e8cec731373f3697c28d67ae079" exitCode=0 Nov 03 00:23:50.978319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:50.977881 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" event={"ID":"2a5e3786-63f4-4a67-910c-53320c5e1879","Type":"ContainerDied","Data":"f9a3a5d8073634c4711f223bfbe54622fc778e8cec731373f3697c28d67ae079"} Nov 03 00:23:52.153582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.153561 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:52.275723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.275698 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr\" (UniqueName: \"kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr\") pod \"2a5e3786-63f4-4a67-910c-53320c5e1879\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " Nov 03 00:23:52.275904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.275789 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca\") pod \"2a5e3786-63f4-4a67-910c-53320c5e1879\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " Nov 03 00:23:52.275904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.275827 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca\") pod \"2a5e3786-63f4-4a67-910c-53320c5e1879\" (UID: \"2a5e3786-63f4-4a67-910c-53320c5e1879\") " Nov 03 00:23:52.276179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.276155 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca" pod "2a5e3786-63f4-4a67-910c-53320c5e1879" (UID: "2a5e3786-63f4-4a67-910c-53320c5e1879"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:23:52.278554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.278521 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca" pod "2a5e3786-63f4-4a67-910c-53320c5e1879" (UID: "2a5e3786-63f4-4a67-910c-53320c5e1879"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:23:52.278697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.278600 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr" pod "2a5e3786-63f4-4a67-910c-53320c5e1879" (UID: "2a5e3786-63f4-4a67-910c-53320c5e1879"). InnerVolumeSpecName "kube-api-access-d7jzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:23:52.377812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.377785 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:52.377812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.377810 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a5e3786-63f4-4a67-910c-53320c5e1879-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:52.377955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.377821 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7jzr\" (UniqueName: \"kubernetes.io/projected/2a5e3786-63f4-4a67-910c-53320c5e1879-kube-api-access-d7jzr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:23:52.993264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.993233 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" Nov 03 00:23:52.993427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.993240 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql" event={"ID":"2a5e3786-63f4-4a67-910c-53320c5e1879","Type":"ContainerDied","Data":"7cbc662af8aed8e9be7e6a1eaf97553ec9825fdc747f2ac5a34144030d09210b"} Nov 03 00:23:52.993427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:52.993361 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7cbc662af8aed8e9be7e6a1eaf97553ec9825fdc747f2ac5a34144030d09210b" Nov 03 00:23:59.509358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.509324 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl"] Nov 03 00:23:59.510135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.510114 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2a5e3786-63f4-4a67-910c-53320c5e1879" containerName="revoke-keys" Nov 03 00:23:59.510236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.510136 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a5e3786-63f4-4a67-910c-53320c5e1879" containerName="revoke-keys" Nov 03 00:23:59.510236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.510155 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1e0a72fc-f718-4e55-ab56-0f62f4bae131" containerName="revoke-keys" Nov 03 00:23:59.510236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.510162 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0a72fc-f718-4e55-ab56-0f62f4bae131" containerName="revoke-keys" Nov 03 00:23:59.510396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.510297 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a5e3786-63f4-4a67-910c-53320c5e1879" containerName="revoke-keys" Nov 03 00:23:59.510396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.510313 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0a72fc-f718-4e55-ab56-0f62f4bae131" containerName="revoke-keys" Nov 03 00:23:59.514766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.514746 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.571831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.571798 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl"] Nov 03 00:23:59.615569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.615443 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:23:59.616831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.616796 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:23:59.641670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.639735 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqjhc\" (UniqueName: \"kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.641670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.639915 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.641670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.640039 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.741603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.741563 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqjhc\" (UniqueName: \"kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.741796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.741726 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.741863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.741840 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.742357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.742314 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.744259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.744232 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.804741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.804665 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqjhc\" (UniqueName: \"kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc\") pod \"maas-api-revoke-sub-db523f03-tr6zl\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:23:59.878658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:23:59.872871 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:24:00.262246 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:00.262220 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598ed79f_b24b_4ba9_ab94_a4064b0630d8.slice/crio-a47e07a93198bf4606d30cd27357a42717221da847938e481db31e2169761545 WatchSource:0}: Error finding container a47e07a93198bf4606d30cd27357a42717221da847938e481db31e2169761545: Status 404 returned error can't find the container with id a47e07a93198bf4606d30cd27357a42717221da847938e481db31e2169761545 Nov 03 00:24:00.262878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:00.262855 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl"] Nov 03 00:24:01.076265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.076229 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" event={"ID":"598ed79f-b24b-4ba9-ab94-a4064b0630d8","Type":"ContainerStarted","Data":"317bc3032035ee6dbde6f2c5f8254da90d09c73f62fa704c652a019c2b3a1d3f"} Nov 03 00:24:01.076685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.076291 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" event={"ID":"598ed79f-b24b-4ba9-ab94-a4064b0630d8","Type":"ContainerStarted","Data":"a47e07a93198bf4606d30cd27357a42717221da847938e481db31e2169761545"} Nov 03 00:24:01.193337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.193289 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" podStartSLOduration=2.193276445 podStartE2EDuration="2.193276445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:01.191299807 +0000 UTC m=+1131.336994273" watchObservedRunningTime="2026-11-03 00:24:01.193276445 +0000 UTC m=+1131.338970935" Nov 03 00:24:01.881520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.881486 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc"] Nov 03 00:24:01.885477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.885461 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:01.942254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.942222 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc"] Nov 03 00:24:01.967518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.967491 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:01.967700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.967678 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p222g\" (UniqueName: \"kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:01.967874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:01.967856 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.069004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.068977 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.069149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.069041 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.069149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.069105 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p222g\" (UniqueName: \"kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.069679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.069649 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.071466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.071442 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.085921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.085898 5144 generic.go:356] "Generic (PLEG): container finished" podID="598ed79f-b24b-4ba9-ab94-a4064b0630d8" containerID="317bc3032035ee6dbde6f2c5f8254da90d09c73f62fa704c652a019c2b3a1d3f" exitCode=0 Nov 03 00:24:02.086282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.085980 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" event={"ID":"598ed79f-b24b-4ba9-ab94-a4064b0630d8","Type":"ContainerDied","Data":"317bc3032035ee6dbde6f2c5f8254da90d09c73f62fa704c652a019c2b3a1d3f"} Nov 03 00:24:02.125879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.125848 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p222g\" (UniqueName: \"kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g\") pod \"maas-api-revoke-sub-293a3dfd-m5tqc\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.198786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.198710 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:02.589266 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:02.589236 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de2c310_0020_48f5_a718_a2ab673e0f65.slice/crio-c3c4fa8002a6c290e30598d98e8379a7f9d22da5a539756f4e65a64ae306410b WatchSource:0}: Error finding container c3c4fa8002a6c290e30598d98e8379a7f9d22da5a539756f4e65a64ae306410b: Status 404 returned error can't find the container with id c3c4fa8002a6c290e30598d98e8379a7f9d22da5a539756f4e65a64ae306410b Nov 03 00:24:02.590839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:02.590808 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc"] Nov 03 00:24:03.094472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.094440 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" event={"ID":"7de2c310-0020-48f5-a718-a2ab673e0f65","Type":"ContainerStarted","Data":"ea4d32534dfe71dca6326e5f8cc06634200875b8dabeb497443c16af643b037a"} Nov 03 00:24:03.094949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.094488 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" event={"ID":"7de2c310-0020-48f5-a718-a2ab673e0f65","Type":"ContainerStarted","Data":"c3c4fa8002a6c290e30598d98e8379a7f9d22da5a539756f4e65a64ae306410b"} Nov 03 00:24:03.213503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.213413 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" podStartSLOduration=2.213398649 podStartE2EDuration="2.213398649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:03.210578074 +0000 UTC m=+1133.356272553" watchObservedRunningTime="2026-11-03 00:24:03.213398649 +0000 UTC m=+1133.359093149" Nov 03 00:24:03.256174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.256149 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:24:03.386554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.386524 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca\") pod \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " Nov 03 00:24:03.386757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.386606 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc\" (UniqueName: \"kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc\") pod \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " Nov 03 00:24:03.386757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.386691 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca\") pod \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\" (UID: \"598ed79f-b24b-4ba9-ab94-a4064b0630d8\") " Nov 03 00:24:03.386977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.386945 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca" pod "598ed79f-b24b-4ba9-ab94-a4064b0630d8" (UID: "598ed79f-b24b-4ba9-ab94-a4064b0630d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:03.387128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.387098 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:03.389534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.389509 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc" pod "598ed79f-b24b-4ba9-ab94-a4064b0630d8" (UID: "598ed79f-b24b-4ba9-ab94-a4064b0630d8"). InnerVolumeSpecName "kube-api-access-qqjhc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:03.389624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.389583 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca" pod "598ed79f-b24b-4ba9-ab94-a4064b0630d8" (UID: "598ed79f-b24b-4ba9-ab94-a4064b0630d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:03.488499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.488420 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqjhc\" (UniqueName: \"kubernetes.io/projected/598ed79f-b24b-4ba9-ab94-a4064b0630d8-kube-api-access-qqjhc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:03.488499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:03.488453 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598ed79f-b24b-4ba9-ab94-a4064b0630d8-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:04.102459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:04.102433 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" Nov 03 00:24:04.102873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:04.102473 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl" event={"ID":"598ed79f-b24b-4ba9-ab94-a4064b0630d8","Type":"ContainerDied","Data":"a47e07a93198bf4606d30cd27357a42717221da847938e481db31e2169761545"} Nov 03 00:24:04.102873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:04.102523 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47e07a93198bf4606d30cd27357a42717221da847938e481db31e2169761545" Nov 03 00:24:04.104295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:04.104265 5144 generic.go:356] "Generic (PLEG): container finished" podID="7de2c310-0020-48f5-a718-a2ab673e0f65" containerID="ea4d32534dfe71dca6326e5f8cc06634200875b8dabeb497443c16af643b037a" exitCode=0 Nov 03 00:24:04.104424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:04.104299 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" event={"ID":"7de2c310-0020-48f5-a718-a2ab673e0f65","Type":"ContainerDied","Data":"ea4d32534dfe71dca6326e5f8cc06634200875b8dabeb497443c16af643b037a"} Nov 03 00:24:05.246568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.246547 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:05.410996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.410924 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca\") pod \"7de2c310-0020-48f5-a718-a2ab673e0f65\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " Nov 03 00:24:05.410996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.410964 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca\") pod \"7de2c310-0020-48f5-a718-a2ab673e0f65\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " Nov 03 00:24:05.411208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.411081 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g\" (UniqueName: \"kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g\") pod \"7de2c310-0020-48f5-a718-a2ab673e0f65\" (UID: \"7de2c310-0020-48f5-a718-a2ab673e0f65\") " Nov 03 00:24:05.411391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.411367 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca" pod "7de2c310-0020-48f5-a718-a2ab673e0f65" (UID: "7de2c310-0020-48f5-a718-a2ab673e0f65"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:05.411550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.411536 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:05.413212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.413191 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca" pod "7de2c310-0020-48f5-a718-a2ab673e0f65" (UID: "7de2c310-0020-48f5-a718-a2ab673e0f65"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:05.413339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.413320 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g" pod "7de2c310-0020-48f5-a718-a2ab673e0f65" (UID: "7de2c310-0020-48f5-a718-a2ab673e0f65"). InnerVolumeSpecName "kube-api-access-p222g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:05.512330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.512303 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de2c310-0020-48f5-a718-a2ab673e0f65-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:05.512330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:05.512326 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p222g\" (UniqueName: \"kubernetes.io/projected/7de2c310-0020-48f5-a718-a2ab673e0f65-kube-api-access-p222g\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:06.057173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:06.057142 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r"] Nov 03 00:24:06.109387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:06.109352 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a89a9d9-dcb7r"] Nov 03 00:24:06.120857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:06.120830 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" Nov 03 00:24:06.120857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:06.120841 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc" event={"ID":"7de2c310-0020-48f5-a718-a2ab673e0f65","Type":"ContainerDied","Data":"c3c4fa8002a6c290e30598d98e8379a7f9d22da5a539756f4e65a64ae306410b"} Nov 03 00:24:06.121015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:06.120905 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c3c4fa8002a6c290e30598d98e8379a7f9d22da5a539756f4e65a64ae306410b" Nov 03 00:24:07.996455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:07.996426 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ef5712-612b-47d9-857c-63fc93dbbfba" path="/var/lib/kubelet/pods/93ef5712-612b-47d9-857c-63fc93dbbfba/volumes" Nov 03 00:24:12.155356 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:24:12.155320 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:24:16.646219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646186 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974"] Nov 03 00:24:16.646776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646761 5144 cpu_manager.go:414] "containerMap: removing container" podUID="598ed79f-b24b-4ba9-ab94-a4064b0630d8" containerName="revoke-keys" Nov 03 00:24:16.646776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646777 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598ed79f-b24b-4ba9-ab94-a4064b0630d8" containerName="revoke-keys" Nov 03 00:24:16.646878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646800 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7de2c310-0020-48f5-a718-a2ab673e0f65" containerName="revoke-keys" Nov 03 00:24:16.646878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646805 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de2c310-0020-48f5-a718-a2ab673e0f65" containerName="revoke-keys" Nov 03 00:24:16.646941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646898 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="598ed79f-b24b-4ba9-ab94-a4064b0630d8" containerName="revoke-keys" Nov 03 00:24:16.646941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.646905 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de2c310-0020-48f5-a718-a2ab673e0f65" containerName="revoke-keys" Nov 03 00:24:16.651531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.651515 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.705039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.705014 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974"] Nov 03 00:24:16.750106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.750075 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:24:16.750217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.750121 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:24:16.812091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.812059 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rzrt\" (UniqueName: \"kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.812264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.812239 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.812387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.812292 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.913858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.913794 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.913858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.913850 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rzrt\" (UniqueName: \"kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.914049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.913965 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.914432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.914411 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.916484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.916465 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:16.970358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:16.970329 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rzrt\" (UniqueName: \"kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt\") pod \"maas-api-revoke-sub-0b96e18f-l9974\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:17.264767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:17.264744 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:17.644968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:17.644941 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974"] Nov 03 00:24:17.645851 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:17.645822 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f48a828_89b9_4c39_b6ea_7c9d5577bd0a.slice/crio-92cedf49583bf2980afebfdf70cccafe76de5b287e47055d6f4d0f3809318ef0 WatchSource:0}: Error finding container 92cedf49583bf2980afebfdf70cccafe76de5b287e47055d6f4d0f3809318ef0: Status 404 returned error can't find the container with id 92cedf49583bf2980afebfdf70cccafe76de5b287e47055d6f4d0f3809318ef0 Nov 03 00:24:18.232319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:18.232237 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" event={"ID":"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a","Type":"ContainerStarted","Data":"c5ca7c13564712f582491ad5d7df5a7761db4cfefdf820ec448f8d91d3c72b47"} Nov 03 00:24:18.232319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:18.232291 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" event={"ID":"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a","Type":"ContainerStarted","Data":"92cedf49583bf2980afebfdf70cccafe76de5b287e47055d6f4d0f3809318ef0"} Nov 03 00:24:18.345699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:18.345650 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" podStartSLOduration=2.345617474 podStartE2EDuration="2.345617474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:18.345271484 +0000 UTC m=+1148.490965959" watchObservedRunningTime="2026-11-03 00:24:18.345617474 +0000 UTC m=+1148.491311929" Nov 03 00:24:19.241237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.241201 5144 generic.go:356] "Generic (PLEG): container finished" podID="3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" containerID="c5ca7c13564712f582491ad5d7df5a7761db4cfefdf820ec448f8d91d3c72b47" exitCode=0 Nov 03 00:24:19.241703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.241288 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" event={"ID":"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a","Type":"ContainerDied","Data":"c5ca7c13564712f582491ad5d7df5a7761db4cfefdf820ec448f8d91d3c72b47"} Nov 03 00:24:19.810753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.810724 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8"] Nov 03 00:24:19.815128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.815111 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:19.871645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.871601 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8"] Nov 03 00:24:19.951475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.951443 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:19.951661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.951524 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9s98\" (UniqueName: \"kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:19.951661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:19.951585 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.052253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.052220 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.052423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.052309 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9s98\" (UniqueName: \"kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.052423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.052353 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.052827 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.052802 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.054902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.054876 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.108306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.108235 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9s98\" (UniqueName: \"kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98\") pod \"maas-api-revoke-sub-fc92b52a-cjvz8\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.129119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.129092 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:20.379719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.379691 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:20.515384 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:20.515335 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44abe098_1c18_4929_89e6_24ead8738985.slice/crio-4c8359e525e5fcc523fdb088b43c5d8ef2e85de8742cde53638f55a049ea10e5 WatchSource:0}: Error finding container 4c8359e525e5fcc523fdb088b43c5d8ef2e85de8742cde53638f55a049ea10e5: Status 404 returned error can't find the container with id 4c8359e525e5fcc523fdb088b43c5d8ef2e85de8742cde53638f55a049ea10e5 Nov 03 00:24:20.545981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.545954 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8"] Nov 03 00:24:20.558734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.558698 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca\") pod \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " Nov 03 00:24:20.558845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.558750 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca\") pod \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " Nov 03 00:24:20.558915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.558867 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt\" (UniqueName: \"kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt\") pod \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\" (UID: \"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a\") " Nov 03 00:24:20.559070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.559044 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca" pod "3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" (UID: "3f48a828-89b9-4c39-b6ea-7c9d5577bd0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:20.559356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.559332 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:20.561019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.560999 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca" pod "3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" (UID: "3f48a828-89b9-4c39-b6ea-7c9d5577bd0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:20.561470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.561445 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt" pod "3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" (UID: "3f48a828-89b9-4c39-b6ea-7c9d5577bd0a"). InnerVolumeSpecName "kube-api-access-2rzrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:20.660535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.660459 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:20.660535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:20.660483 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rzrt\" (UniqueName: \"kubernetes.io/projected/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a-kube-api-access-2rzrt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:21.258971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:21.258941 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" event={"ID":"3f48a828-89b9-4c39-b6ea-7c9d5577bd0a","Type":"ContainerDied","Data":"92cedf49583bf2980afebfdf70cccafe76de5b287e47055d6f4d0f3809318ef0"} Nov 03 00:24:21.259183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:21.258999 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92cedf49583bf2980afebfdf70cccafe76de5b287e47055d6f4d0f3809318ef0" Nov 03 00:24:21.259183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:21.258950 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974" Nov 03 00:24:21.260860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:21.260832 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" event={"ID":"44abe098-1c18-4929-89e6-24ead8738985","Type":"ContainerStarted","Data":"7fa7a2875679881686a58e8f0db00f6bcdae8309611e13bf3c498c29ec9b48e5"} Nov 03 00:24:21.261021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:21.260892 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" event={"ID":"44abe098-1c18-4929-89e6-24ead8738985","Type":"ContainerStarted","Data":"4c8359e525e5fcc523fdb088b43c5d8ef2e85de8742cde53638f55a049ea10e5"} Nov 03 00:24:21.377038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:21.376996 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" podStartSLOduration=2.376983308 podStartE2EDuration="2.376983308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:21.374106443 +0000 UTC m=+1151.519800911" watchObservedRunningTime="2026-11-03 00:24:21.376983308 +0000 UTC m=+1151.522677815" Nov 03 00:24:22.269563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:22.269538 5144 generic.go:356] "Generic (PLEG): container finished" podID="44abe098-1c18-4929-89e6-24ead8738985" containerID="7fa7a2875679881686a58e8f0db00f6bcdae8309611e13bf3c498c29ec9b48e5" exitCode=0 Nov 03 00:24:22.269964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:22.269622 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" event={"ID":"44abe098-1c18-4929-89e6-24ead8738985","Type":"ContainerDied","Data":"7fa7a2875679881686a58e8f0db00f6bcdae8309611e13bf3c498c29ec9b48e5"} Nov 03 00:24:23.046922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.046882 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6"] Nov 03 00:24:23.096117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.096089 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fc9ead0-6cdd6"] Nov 03 00:24:23.433142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.433112 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:23.501372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.501346 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca\") pod \"44abe098-1c18-4929-89e6-24ead8738985\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " Nov 03 00:24:23.501489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.501436 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca\") pod \"44abe098-1c18-4929-89e6-24ead8738985\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " Nov 03 00:24:23.501489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.501459 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98\" (UniqueName: \"kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98\") pod \"44abe098-1c18-4929-89e6-24ead8738985\" (UID: \"44abe098-1c18-4929-89e6-24ead8738985\") " Nov 03 00:24:23.501761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.501737 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca" pod "44abe098-1c18-4929-89e6-24ead8738985" (UID: "44abe098-1c18-4929-89e6-24ead8738985"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:23.503645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.503599 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca" pod "44abe098-1c18-4929-89e6-24ead8738985" (UID: "44abe098-1c18-4929-89e6-24ead8738985"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:23.503741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.503661 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98" pod "44abe098-1c18-4929-89e6-24ead8738985" (UID: "44abe098-1c18-4929-89e6-24ead8738985"). InnerVolumeSpecName "kube-api-access-c9s98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:23.603162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.603092 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44abe098-1c18-4929-89e6-24ead8738985-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:23.603162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.603116 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9s98\" (UniqueName: \"kubernetes.io/projected/44abe098-1c18-4929-89e6-24ead8738985-kube-api-access-c9s98\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:23.603162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.603127 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44abe098-1c18-4929-89e6-24ead8738985-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:23.995992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:23.995903 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ad5e6a3-aa92-4dc5-9a27-0133833f93ad" path="/var/lib/kubelet/pods/8ad5e6a3-aa92-4dc5-9a27-0133833f93ad/volumes" Nov 03 00:24:24.177040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.177002 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82"] Nov 03 00:24:24.177887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.177854 5144 cpu_manager.go:414] "containerMap: removing container" podUID="44abe098-1c18-4929-89e6-24ead8738985" containerName="revoke-keys" Nov 03 00:24:24.177887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.177892 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44abe098-1c18-4929-89e6-24ead8738985" containerName="revoke-keys" Nov 03 00:24:24.177999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.177970 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" containerName="revoke-keys" Nov 03 00:24:24.177999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.177979 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" containerName="revoke-keys" Nov 03 00:24:24.178129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.178116 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="44abe098-1c18-4929-89e6-24ead8738985" containerName="revoke-keys" Nov 03 00:24:24.178163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.178141 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" containerName="revoke-keys" Nov 03 00:24:24.182032 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.182018 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209066 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209138 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209186 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209210 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209245 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209270 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209299 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209379 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.209458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.209406 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr6rj\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.292436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.292406 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82"] Nov 03 00:24:24.293471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.293440 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" event={"ID":"44abe098-1c18-4929-89e6-24ead8738985","Type":"ContainerDied","Data":"4c8359e525e5fcc523fdb088b43c5d8ef2e85de8742cde53638f55a049ea10e5"} Nov 03 00:24:24.293573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.293497 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4c8359e525e5fcc523fdb088b43c5d8ef2e85de8742cde53638f55a049ea10e5" Nov 03 00:24:24.293573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.293562 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8" Nov 03 00:24:24.310812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.310774 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.310943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.310841 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.310943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.310931 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.310977 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr6rj\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311043 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311142 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311167 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311193 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311218 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311533 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311738 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.311912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.311274 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.312191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.312166 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.312300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.312280 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.313876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.313852 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.314366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.314346 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.367160 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.367134 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.367341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.367320 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr6rj\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.495268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.495233 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:24.927504 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:24.927468 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88c37e5_26f1_453a_8e25_4f288c2b8c24.slice/crio-81c97a8619678c08e3915a0824d449b4361de4e85bed559cb3f7d9b6c8fee4cc WatchSource:0}: Error finding container 81c97a8619678c08e3915a0824d449b4361de4e85bed559cb3f7d9b6c8fee4cc: Status 404 returned error can't find the container with id 81c97a8619678c08e3915a0824d449b4361de4e85bed559cb3f7d9b6c8fee4cc Nov 03 00:24:24.928922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.928895 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82"] Nov 03 00:24:24.930163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.930134 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:24.930260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.930192 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:24.930260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:24.930221 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:25.304285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:25.304247 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" event={"ID":"e88c37e5-26f1-453a-8e25-4f288c2b8c24","Type":"ContainerStarted","Data":"a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d"} Nov 03 00:24:25.304549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:25.304523 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" event={"ID":"e88c37e5-26f1-453a-8e25-4f288c2b8c24","Type":"ContainerStarted","Data":"81c97a8619678c08e3915a0824d449b4361de4e85bed559cb3f7d9b6c8fee4cc"} Nov 03 00:24:25.426452 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:25.426082 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podStartSLOduration=1.426067414 podStartE2EDuration="1.426067414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:25.423070101 +0000 UTC m=+1155.568764570" watchObservedRunningTime="2026-11-03 00:24:25.426067414 +0000 UTC m=+1155.571761871" Nov 03 00:24:25.495348 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:25.495313 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:26.501700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:26.501663 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:24:26.502301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:26.502269 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 00:24:27.441063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.441034 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc"] Nov 03 00:24:27.445273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.445254 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.484093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.484072 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc"] Nov 03 00:24:27.496258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.496232 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:24:27.496386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.496284 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 00:24:27.540477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.540449 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:24:27.540982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.540833 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:24:27.541888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.541717 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qprnm\" (UniqueName: \"kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.541888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.541823 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.542130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.542098 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.644681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.643936 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.644681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.644034 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qprnm\" (UniqueName: \"kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.644681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.644124 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.644917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.644775 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.647029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.646996 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.705325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.705261 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qprnm\" (UniqueName: \"kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm\") pod \"maas-api-revoke-sub-d2b087ae-z98lc\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:27.761320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:27.761278 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:28.168674 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:28.164498 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9fcf5a_97ef_4f61_8b84_e8709230984c.slice/crio-dc575d50553bad27f7baabc23ce00b011f31be3f1313df9c2af71d70b46af965 WatchSource:0}: Error finding container dc575d50553bad27f7baabc23ce00b011f31be3f1313df9c2af71d70b46af965: Status 404 returned error can't find the container with id dc575d50553bad27f7baabc23ce00b011f31be3f1313df9c2af71d70b46af965 Nov 03 00:24:28.168674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:28.167191 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc"] Nov 03 00:24:28.354242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:28.354205 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" event={"ID":"0f9fcf5a-97ef-4f61-8b84-e8709230984c","Type":"ContainerStarted","Data":"dc575d50553bad27f7baabc23ce00b011f31be3f1313df9c2af71d70b46af965"} Nov 03 00:24:28.495737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:28.495664 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:24:28.495737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:28.495712 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 00:24:29.363645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.363604 5144 generic.go:356] "Generic (PLEG): container finished" podID="0f9fcf5a-97ef-4f61-8b84-e8709230984c" containerID="87fcbc8b19298d029da56362bc414f84e7cb24f52c489382b4f4d63de358fe7b" exitCode=0 Nov 03 00:24:29.364106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.363700 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" event={"ID":"0f9fcf5a-97ef-4f61-8b84-e8709230984c","Type":"ContainerDied","Data":"87fcbc8b19298d029da56362bc414f84e7cb24f52c489382b4f4d63de358fe7b"} Nov 03 00:24:29.496126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.496050 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:24:29.496126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.496102 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 00:24:29.775165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.775136 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf"] Nov 03 00:24:29.779382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.779363 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.856990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.856951 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf"] Nov 03 00:24:29.878133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.878108 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7c7tc\" (UniqueName: \"kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.878278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.878151 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.878325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.878302 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.979158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.979125 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.979326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.979236 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.979326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.979316 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7c7tc\" (UniqueName: \"kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.979916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.979887 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:29.981509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:29.981486 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:30.038366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.038283 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7c7tc\" (UniqueName: \"kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc\") pod \"maas-api-revoke-sub-77b65026-tdfpf\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:30.093150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.093123 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:30.495837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.495798 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:24:30.496156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.495955 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 00:24:30.505736 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:30.505047 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f45e65_68df_4263_9850_772341980e3e.slice/crio-677a0536cd2afe5cb370eb1621843bb22efa0ba286c0e821358c68c789341b2b WatchSource:0}: Error finding container 677a0536cd2afe5cb370eb1621843bb22efa0ba286c0e821358c68c789341b2b: Status 404 returned error can't find the container with id 677a0536cd2afe5cb370eb1621843bb22efa0ba286c0e821358c68c789341b2b Nov 03 00:24:30.505736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.505441 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf"] Nov 03 00:24:30.591101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.591080 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:30.611995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.611972 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw"] Nov 03 00:24:30.612568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.612554 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0f9fcf5a-97ef-4f61-8b84-e8709230984c" containerName="revoke-keys" Nov 03 00:24:30.612615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.612569 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9fcf5a-97ef-4f61-8b84-e8709230984c" containerName="revoke-keys" Nov 03 00:24:30.612693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.612682 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9fcf5a-97ef-4f61-8b84-e8709230984c" containerName="revoke-keys" Nov 03 00:24:30.648691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.647896 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.672325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.672298 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw"] Nov 03 00:24:30.688575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.688554 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca\") pod \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " Nov 03 00:24:30.688700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.688645 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca\") pod \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " Nov 03 00:24:30.688746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.688728 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm\" (UniqueName: \"kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm\") pod \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\" (UID: \"0f9fcf5a-97ef-4f61-8b84-e8709230984c\") " Nov 03 00:24:30.688988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.688971 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.689050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.689019 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.689050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.689038 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8zzk\" (UniqueName: \"kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.689175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.689158 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca" pod "0f9fcf5a-97ef-4f61-8b84-e8709230984c" (UID: "0f9fcf5a-97ef-4f61-8b84-e8709230984c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:30.691396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.691317 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca" pod "0f9fcf5a-97ef-4f61-8b84-e8709230984c" (UID: "0f9fcf5a-97ef-4f61-8b84-e8709230984c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:30.691619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.691598 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm" pod "0f9fcf5a-97ef-4f61-8b84-e8709230984c" (UID: "0f9fcf5a-97ef-4f61-8b84-e8709230984c"). InnerVolumeSpecName "kube-api-access-qprnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:30.790297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790267 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.790484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790317 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.790484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790348 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8zzk\" (UniqueName: \"kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.790484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790419 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:30.790484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790435 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qprnm\" (UniqueName: \"kubernetes.io/projected/0f9fcf5a-97ef-4f61-8b84-e8709230984c-kube-api-access-qprnm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:30.790484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790449 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f9fcf5a-97ef-4f61-8b84-e8709230984c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:30.790913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.790893 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.792791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.792765 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.870558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.870498 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8zzk\" (UniqueName: \"kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk\") pod \"maas-api-revoke-sub-9dda34b1-w2bvw\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:30.962697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:30.962603 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:31.358925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.354566 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw"] Nov 03 00:24:31.390924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.390888 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" event={"ID":"0f9fcf5a-97ef-4f61-8b84-e8709230984c","Type":"ContainerDied","Data":"dc575d50553bad27f7baabc23ce00b011f31be3f1313df9c2af71d70b46af965"} Nov 03 00:24:31.391045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.390956 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc575d50553bad27f7baabc23ce00b011f31be3f1313df9c2af71d70b46af965" Nov 03 00:24:31.391045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.391034 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc" Nov 03 00:24:31.396111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.396030 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" event={"ID":"f1f45e65-68df-4263-9850-772341980e3e","Type":"ContainerStarted","Data":"c1664286bc052cfb8a535305cb3ca9ce468c543468463f754bf5149a93c37885"} Nov 03 00:24:31.396111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.396078 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" event={"ID":"f1f45e65-68df-4263-9850-772341980e3e","Type":"ContainerStarted","Data":"677a0536cd2afe5cb370eb1621843bb22efa0ba286c0e821358c68c789341b2b"} Nov 03 00:24:31.398312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.398286 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" event={"ID":"e7cec7c9-baf6-473b-afad-530986351553","Type":"ContainerStarted","Data":"036bc0e88f7b263751c60003e9ba3927a324eb2811e34528b11da59c2ce88cd5"} Nov 03 00:24:31.496425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.496389 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 00:24:31.496796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.496457 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 00:24:31.510814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:31.510771 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" podStartSLOduration=2.510756438 podStartE2EDuration="2.510756438s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:31.508388663 +0000 UTC m=+1161.654083120" watchObservedRunningTime="2026-11-03 00:24:31.510756438 +0000 UTC m=+1161.656450894" Nov 03 00:24:32.406832 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.406799 5144 generic.go:356] "Generic (PLEG): container finished" podID="f1f45e65-68df-4263-9850-772341980e3e" containerID="c1664286bc052cfb8a535305cb3ca9ce468c543468463f754bf5149a93c37885" exitCode=0 Nov 03 00:24:32.407004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.406887 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" event={"ID":"f1f45e65-68df-4263-9850-772341980e3e","Type":"ContainerDied","Data":"c1664286bc052cfb8a535305cb3ca9ce468c543468463f754bf5149a93c37885"} Nov 03 00:24:32.408522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.408495 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" event={"ID":"e7cec7c9-baf6-473b-afad-530986351553","Type":"ContainerStarted","Data":"f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4"} Nov 03 00:24:32.443918 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:24:32.443888 5144 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cec7c9_baf6_473b_afad_530986351553.slice/crio-conmon-f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cec7c9_baf6_473b_afad_530986351553.slice/crio-f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:24:32.444018 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:24:32.443965 5144 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cec7c9_baf6_473b_afad_530986351553.slice/crio-f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7cec7c9_baf6_473b_afad_530986351553.slice/crio-conmon-f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:24:32.499775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.499749 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:32.500090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.500027 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:32.500721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.500704 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:24:32.633244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:32.633200 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" podStartSLOduration=2.633187469 podStartE2EDuration="2.633187469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:32.631080796 +0000 UTC m=+1162.776775276" watchObservedRunningTime="2026-11-03 00:24:32.633187469 +0000 UTC m=+1162.778881924" Nov 03 00:24:33.417384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.417352 5144 generic.go:356] "Generic (PLEG): container finished" podID="e7cec7c9-baf6-473b-afad-530986351553" containerID="f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4" exitCode=0 Nov 03 00:24:33.417552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.417433 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" event={"ID":"e7cec7c9-baf6-473b-afad-530986351553","Type":"ContainerDied","Data":"f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4"} Nov 03 00:24:33.578214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.578187 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:33.622359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.622328 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc\" (UniqueName: \"kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc\") pod \"f1f45e65-68df-4263-9850-772341980e3e\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " Nov 03 00:24:33.622523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.622394 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca\") pod \"f1f45e65-68df-4263-9850-772341980e3e\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " Nov 03 00:24:33.622523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.622432 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca\") pod \"f1f45e65-68df-4263-9850-772341980e3e\" (UID: \"f1f45e65-68df-4263-9850-772341980e3e\") " Nov 03 00:24:33.622826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.622804 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca" pod "f1f45e65-68df-4263-9850-772341980e3e" (UID: "f1f45e65-68df-4263-9850-772341980e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:33.622942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.622923 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1f45e65-68df-4263-9850-772341980e3e-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:33.625135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.625113 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca" pod "f1f45e65-68df-4263-9850-772341980e3e" (UID: "f1f45e65-68df-4263-9850-772341980e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:33.625196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.625144 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc" pod "f1f45e65-68df-4263-9850-772341980e3e" (UID: "f1f45e65-68df-4263-9850-772341980e3e"). InnerVolumeSpecName "kube-api-access-7c7tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:33.725129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.725048 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1f45e65-68df-4263-9850-772341980e3e-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:33.725129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:33.725075 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7c7tc\" (UniqueName: \"kubernetes.io/projected/f1f45e65-68df-4263-9850-772341980e3e-kube-api-access-7c7tc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:34.427830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.427752 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" Nov 03 00:24:34.427830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.427765 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf" event={"ID":"f1f45e65-68df-4263-9850-772341980e3e","Type":"ContainerDied","Data":"677a0536cd2afe5cb370eb1621843bb22efa0ba286c0e821358c68c789341b2b"} Nov 03 00:24:34.427830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.427816 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="677a0536cd2afe5cb370eb1621843bb22efa0ba286c0e821358c68c789341b2b" Nov 03 00:24:34.606810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.606784 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:34.743421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.743341 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca\") pod \"e7cec7c9-baf6-473b-afad-530986351553\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " Nov 03 00:24:34.743589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.743559 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk\" (UniqueName: \"kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk\") pod \"e7cec7c9-baf6-473b-afad-530986351553\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " Nov 03 00:24:34.743675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.743619 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca\") pod \"e7cec7c9-baf6-473b-afad-530986351553\" (UID: \"e7cec7c9-baf6-473b-afad-530986351553\") " Nov 03 00:24:34.744138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.744110 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca" pod "e7cec7c9-baf6-473b-afad-530986351553" (UID: "e7cec7c9-baf6-473b-afad-530986351553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:34.750675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.746018 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca" pod "e7cec7c9-baf6-473b-afad-530986351553" (UID: "e7cec7c9-baf6-473b-afad-530986351553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:34.750675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.746317 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk" pod "e7cec7c9-baf6-473b-afad-530986351553" (UID: "e7cec7c9-baf6-473b-afad-530986351553"). InnerVolumeSpecName "kube-api-access-f8zzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:34.844969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.844943 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e7cec7c9-baf6-473b-afad-530986351553-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:34.844969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.844967 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e7cec7c9-baf6-473b-afad-530986351553-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:34.845157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:34.844976 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8zzk\" (UniqueName: \"kubernetes.io/projected/e7cec7c9-baf6-473b-afad-530986351553-kube-api-access-f8zzk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:35.437094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:35.437065 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" Nov 03 00:24:35.437268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:35.437101 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw" event={"ID":"e7cec7c9-baf6-473b-afad-530986351553","Type":"ContainerDied","Data":"036bc0e88f7b263751c60003e9ba3927a324eb2811e34528b11da59c2ce88cd5"} Nov 03 00:24:35.437268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:35.437155 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="036bc0e88f7b263751c60003e9ba3927a324eb2811e34528b11da59c2ce88cd5" Nov 03 00:24:38.677060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677026 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh"] Nov 03 00:24:38.677586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677573 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e7cec7c9-baf6-473b-afad-530986351553" containerName="revoke-keys" Nov 03 00:24:38.677651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677587 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7cec7c9-baf6-473b-afad-530986351553" containerName="revoke-keys" Nov 03 00:24:38.677651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677641 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f1f45e65-68df-4263-9850-772341980e3e" containerName="revoke-keys" Nov 03 00:24:38.677651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677646 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f45e65-68df-4263-9850-772341980e3e" containerName="revoke-keys" Nov 03 00:24:38.677752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677743 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7cec7c9-baf6-473b-afad-530986351553" containerName="revoke-keys" Nov 03 00:24:38.677789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.677758 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f45e65-68df-4263-9850-772341980e3e" containerName="revoke-keys" Nov 03 00:24:38.681433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.681415 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.783783 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.783755 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh"] Nov 03 00:24:38.788209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788175 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788397 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788594 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788727 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788799 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788841 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6wn\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788919 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.788987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.788946 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.789044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.789013 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890254 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890298 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6wn\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890356 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890373 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890501 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890590 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890618 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890696 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890745 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.890818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890777 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.891021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.890921 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.891097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.891071 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.891151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.891105 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.891275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.891256 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.892753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.892734 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.893180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.893165 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.947154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.947088 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.947258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.947159 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6wn\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-2p9qh\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:38.995036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:38.995010 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:39.309048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.309018 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv"] Nov 03 00:24:39.315521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.315488 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.401657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.401612 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv"] Nov 03 00:24:39.402416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402393 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402454 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402484 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402661 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402723 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402770 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402799 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402828 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlgg6\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.402954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.402894 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.471702 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:39.471675 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6d6d3ec_41d6_4fa5_875b_b30fe3f184eb.slice/crio-0e18e1b851b59c7fe966e02e6bd73e042a228f26f4dc1dd8d16af7e7a7e7309e WatchSource:0}: Error finding container 0e18e1b851b59c7fe966e02e6bd73e042a228f26f4dc1dd8d16af7e7a7e7309e: Status 404 returned error can't find the container with id 0e18e1b851b59c7fe966e02e6bd73e042a228f26f4dc1dd8d16af7e7a7e7309e Nov 03 00:24:39.472004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.471983 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh"] Nov 03 00:24:39.474201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.474170 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:39.474333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.474237 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:39.474333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.474280 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:39.503485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503463 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503508 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503526 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503550 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503611 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503674 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503703 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503737 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlgg6\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503810 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.503988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.503926 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.504412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.504390 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.505192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.504758 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.505192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.504793 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.505192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.504879 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.506159 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.506142 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.506587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.506569 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.560425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.560363 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.560868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.560841 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlgg6\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:39.632169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:39.632138 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:40.067509 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:40.067482 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd6edfa_28b9_4c2d_8983_377ec06c0e29.slice/crio-678b6e1976b37d084ee5c1ce1178a7a5a34f32d6f554a068b166f6b7d37f6f4c WatchSource:0}: Error finding container 678b6e1976b37d084ee5c1ce1178a7a5a34f32d6f554a068b166f6b7d37f6f4c: Status 404 returned error can't find the container with id 678b6e1976b37d084ee5c1ce1178a7a5a34f32d6f554a068b166f6b7d37f6f4c Nov 03 00:24:40.068180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.068155 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv"] Nov 03 00:24:40.070230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.070199 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:40.070308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.070264 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:40.070308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.070289 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:24:40.476531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.476441 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" event={"ID":"ffd6edfa-28b9-4c2d-8983-377ec06c0e29","Type":"ContainerStarted","Data":"e0e160d957d14cae904daa34b81356a180d34b3327611d9f131ae3725259bbbd"} Nov 03 00:24:40.476531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.476493 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" event={"ID":"ffd6edfa-28b9-4c2d-8983-377ec06c0e29","Type":"ContainerStarted","Data":"678b6e1976b37d084ee5c1ce1178a7a5a34f32d6f554a068b166f6b7d37f6f4c"} Nov 03 00:24:40.478181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.478148 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" event={"ID":"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb","Type":"ContainerStarted","Data":"07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4"} Nov 03 00:24:40.478296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.478201 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" event={"ID":"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb","Type":"ContainerStarted","Data":"0e18e1b851b59c7fe966e02e6bd73e042a228f26f4dc1dd8d16af7e7a7e7309e"} Nov 03 00:24:40.594769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.594719 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podStartSLOduration=1.594705669 podStartE2EDuration="1.594705669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:40.593544323 +0000 UTC m=+1170.739238796" watchObservedRunningTime="2026-11-03 00:24:40.594705669 +0000 UTC m=+1170.740400123" Nov 03 00:24:40.637658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.632501 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:40.712362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.712303 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podStartSLOduration=2.712285723 podStartE2EDuration="2.712285723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:40.708339635 +0000 UTC m=+1170.854034130" watchObservedRunningTime="2026-11-03 00:24:40.712285723 +0000 UTC m=+1170.857980181" Nov 03 00:24:40.996329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.996295 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:40.997567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.997532 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 00:24:40.997733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:40.997579 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 00:24:41.641665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:41.640768 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 00:24:41.641665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:41.640858 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 00:24:41.995823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:41.995739 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 00:24:41.996060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:41.996036 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 00:24:42.197428 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:24:42.197388 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:24:42.639667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:42.636799 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 00:24:42.639667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:42.636869 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 00:24:42.996413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:42.996334 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 00:24:42.996782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:42.996442 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 00:24:43.067244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.066624 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg"] Nov 03 00:24:43.101863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.101836 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d806472-5pftg"] Nov 03 00:24:43.166655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.165961 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz"] Nov 03 00:24:43.172020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.171995 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.199230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.199143 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.199371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.199282 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmrm\" (UniqueName: \"kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.199371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.199331 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.221654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.214212 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz"] Nov 03 00:24:43.267657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.266774 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:24:43.267657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.266910 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:24:43.302328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.301135 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmrm\" (UniqueName: \"kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.302328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.301178 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.302328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.301256 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.302328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.301841 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.304396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.304373 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.369373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.369342 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmrm\" (UniqueName: \"kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm\") pod \"maas-api-revoke-sub-c071dcf0-mjrvz\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.485957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.485928 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:43.646652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.644954 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 00:24:43.646652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.645011 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 00:24:43.934705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.926252 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz"] Nov 03 00:24:43.956656 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:43.946788 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ed52a_865d_4338_8280_4c20f2bc527c.slice/crio-a6f49eb8ed300ca3b5f2d48d0429ce0ea5cff27d4cc6c19a0a37ffd3995a35a9 WatchSource:0}: Error finding container a6f49eb8ed300ca3b5f2d48d0429ce0ea5cff27d4cc6c19a0a37ffd3995a35a9: Status 404 returned error can't find the container with id a6f49eb8ed300ca3b5f2d48d0429ce0ea5cff27d4cc6c19a0a37ffd3995a35a9 Nov 03 00:24:43.996910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.995505 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 00:24:43.996910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.995574 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 00:24:43.998592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:43.998568 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5" path="/var/lib/kubelet/pods/fc12acd0-fd1e-43c5-99b8-6c33bd71b4d5/volumes" Nov 03 00:24:44.070654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.070150 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs"] Nov 03 00:24:44.079047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.079014 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3666f734-p6qbs"] Nov 03 00:24:44.525590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.525556 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" event={"ID":"dc2ed52a-865d-4338-8280-4c20f2bc527c","Type":"ContainerStarted","Data":"74eb0efe88d75ab40cad020d5f6a4ace71be35bd756b537e65b581d12c213ac6"} Nov 03 00:24:44.525872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.525848 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" event={"ID":"dc2ed52a-865d-4338-8280-4c20f2bc527c","Type":"ContainerStarted","Data":"a6f49eb8ed300ca3b5f2d48d0429ce0ea5cff27d4cc6c19a0a37ffd3995a35a9"} Nov 03 00:24:44.633066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.632954 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 00:24:44.633323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.633280 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 00:24:44.661687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:44.661609 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" podStartSLOduration=1.6615912430000002 podStartE2EDuration="1.661591243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:44.646585635 +0000 UTC m=+1174.792280092" watchObservedRunningTime="2026-11-03 00:24:44.661591243 +0000 UTC m=+1174.807285701" Nov 03 00:24:45.006828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.006784 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 00:24:45.007231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.006855 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 00:24:45.197112 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:24:45.197076 5144 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ed52a_865d_4338_8280_4c20f2bc527c.slice/crio-conmon-74eb0efe88d75ab40cad020d5f6a4ace71be35bd756b537e65b581d12c213ac6.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:24:45.559387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.559330 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" event={"ID":"dc2ed52a-865d-4338-8280-4c20f2bc527c","Type":"ContainerDied","Data":"74eb0efe88d75ab40cad020d5f6a4ace71be35bd756b537e65b581d12c213ac6"} Nov 03 00:24:45.559619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.559590 5144 generic.go:356] "Generic (PLEG): container finished" podID="dc2ed52a-865d-4338-8280-4c20f2bc527c" containerID="74eb0efe88d75ab40cad020d5f6a4ace71be35bd756b537e65b581d12c213ac6" exitCode=0 Nov 03 00:24:45.633655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.633204 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 00:24:45.633655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.633272 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 00:24:45.995522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.995435 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 00:24:45.995522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.995498 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 00:24:45.997973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:45.997951 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3950b165-6af2-46e7-8a28-35d628d15297" path="/var/lib/kubelet/pods/3950b165-6af2-46e7-8a28-35d628d15297/volumes" Nov 03 00:24:46.641225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.638690 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" start-of-body= Nov 03 00:24:46.641225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.638749 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.119:15021/healthz/ready\": dial tcp 10.132.0.119:15021: connect: connection refused" Nov 03 00:24:46.773267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.773242 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:46.809186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.809159 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca\") pod \"dc2ed52a-865d-4338-8280-4c20f2bc527c\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " Nov 03 00:24:46.809333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.809252 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca\") pod \"dc2ed52a-865d-4338-8280-4c20f2bc527c\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " Nov 03 00:24:46.809333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.809292 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm\" (UniqueName: \"kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm\") pod \"dc2ed52a-865d-4338-8280-4c20f2bc527c\" (UID: \"dc2ed52a-865d-4338-8280-4c20f2bc527c\") " Nov 03 00:24:46.809646 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.809599 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca" pod "dc2ed52a-865d-4338-8280-4c20f2bc527c" (UID: "dc2ed52a-865d-4338-8280-4c20f2bc527c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:46.809769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.809755 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:46.811489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.811465 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm" pod "dc2ed52a-865d-4338-8280-4c20f2bc527c" (UID: "dc2ed52a-865d-4338-8280-4c20f2bc527c"). InnerVolumeSpecName "kube-api-access-mqmrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:46.811575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.811510 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca" pod "dc2ed52a-865d-4338-8280-4c20f2bc527c" (UID: "dc2ed52a-865d-4338-8280-4c20f2bc527c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:46.911405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.911370 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc2ed52a-865d-4338-8280-4c20f2bc527c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:46.911523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:46.911408 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqmrm\" (UniqueName: \"kubernetes.io/projected/dc2ed52a-865d-4338-8280-4c20f2bc527c-kube-api-access-mqmrm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:47.000534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.000505 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:47.000963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.000935 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:47.001657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.001611 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:24:47.579367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.579336 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" Nov 03 00:24:47.579551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.579377 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz" event={"ID":"dc2ed52a-865d-4338-8280-4c20f2bc527c","Type":"ContainerDied","Data":"a6f49eb8ed300ca3b5f2d48d0429ce0ea5cff27d4cc6c19a0a37ffd3995a35a9"} Nov 03 00:24:47.579551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.579426 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6f49eb8ed300ca3b5f2d48d0429ce0ea5cff27d4cc6c19a0a37ffd3995a35a9" Nov 03 00:24:47.637519 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.637491 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:47.637913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.637891 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:47.638523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:47.638503 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 00:24:49.538780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.538747 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw"] Nov 03 00:24:49.539370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.539355 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dc2ed52a-865d-4338-8280-4c20f2bc527c" containerName="revoke-keys" Nov 03 00:24:49.539412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.539372 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2ed52a-865d-4338-8280-4c20f2bc527c" containerName="revoke-keys" Nov 03 00:24:49.539508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.539496 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2ed52a-865d-4338-8280-4c20f2bc527c" containerName="revoke-keys" Nov 03 00:24:49.543201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.543179 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.600145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.600115 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw"] Nov 03 00:24:49.642961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.642919 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:24:49.643813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.643790 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:24:49.647288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.647255 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.647534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.647516 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.648123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.648097 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mm72\" (UniqueName: \"kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.753657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.751779 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mm72\" (UniqueName: \"kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.753657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.751938 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.753657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.752006 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.753657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.753078 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.765102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.765045 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.839177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.839103 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mm72\" (UniqueName: \"kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72\") pod \"maas-api-revoke-sub-56eea33d-25cpw\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:49.867888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:49.867854 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:50.755917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:50.755886 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw"] Nov 03 00:24:51.580657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:51.573921 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.629933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:51.629891 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.677790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:51.657792 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.766656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:51.762550 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" event={"ID":"30df45df-f5e5-4ade-818d-d9acede7795a","Type":"ContainerStarted","Data":"e8a113ab4ae00298cdfd8e87c2a753d70ab0469ce4ef9765f2048edaf25283dd"} Nov 03 00:24:51.842346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:51.842296 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:51.970304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:51.969729 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.009204 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.008717 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:24:52.009204 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.008776 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.028664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.028057 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.582656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.577775 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:24:52.582656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.578515 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:24:52.649814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.642322 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 00:24:52.649814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.645979 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:24:52.675656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.662863 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 00:24:52.675656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.663492 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:24:52.675656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.666772 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:24:52.866373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.865809 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 00:24:52.883649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.878206 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 00:24:52.891353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.891321 5144 generic.go:356] "Generic (PLEG): container finished" podID="30df45df-f5e5-4ade-818d-d9acede7795a" containerID="189d0362c419ca4d20bf05c461c015a3561d808ec715823fcdbd9e9dd487e277" exitCode=0 Nov 03 00:24:52.891689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:52.891622 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" event={"ID":"30df45df-f5e5-4ade-818d-d9acede7795a","Type":"ContainerDied","Data":"189d0362c419ca4d20bf05c461c015a3561d808ec715823fcdbd9e9dd487e277"} Nov 03 00:24:53.895974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:53.895940 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw"] Nov 03 00:24:53.900195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:53.900174 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:53.923102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:53.923063 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:53.923261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:53.923226 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:53.923353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:53.923330 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cvcr\" (UniqueName: \"kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:53.960980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:53.960944 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw"] Nov 03 00:24:54.029360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.028135 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.029360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.028272 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.030612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.030238 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.030612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.028342 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cvcr\" (UniqueName: \"kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.041225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.036264 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.051697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.051672 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc"] Nov 03 00:24:54.074994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.074973 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ab4ae6c-tpkwc"] Nov 03 00:24:54.087020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.086994 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cvcr\" (UniqueName: \"kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr\") pod \"maas-api-revoke-sub-7b29c2fc-khjcw\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.104140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.104119 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:54.131001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.130975 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72\" (UniqueName: \"kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72\") pod \"30df45df-f5e5-4ade-818d-d9acede7795a\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " Nov 03 00:24:54.131167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.131153 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca\") pod \"30df45df-f5e5-4ade-818d-d9acede7795a\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " Nov 03 00:24:54.131207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.131189 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca\") pod \"30df45df-f5e5-4ade-818d-d9acede7795a\" (UID: \"30df45df-f5e5-4ade-818d-d9acede7795a\") " Nov 03 00:24:54.131472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.131446 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca" pod "30df45df-f5e5-4ade-818d-d9acede7795a" (UID: "30df45df-f5e5-4ade-818d-d9acede7795a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:54.131617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.131601 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:54.133067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.133043 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72" pod "30df45df-f5e5-4ade-818d-d9acede7795a" (UID: "30df45df-f5e5-4ade-818d-d9acede7795a"). InnerVolumeSpecName "kube-api-access-8mm72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:54.133564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.133548 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca" pod "30df45df-f5e5-4ade-818d-d9acede7795a" (UID: "30df45df-f5e5-4ade-818d-d9acede7795a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:54.222962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.221733 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:54.232804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.232767 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30df45df-f5e5-4ade-818d-d9acede7795a-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:54.232804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.232800 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mm72\" (UniqueName: \"kubernetes.io/projected/30df45df-f5e5-4ade-818d-d9acede7795a-kube-api-access-8mm72\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:54.608838 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:24:54.608810 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9661b07_ed83_4ec4_9948_762b602dff13.slice/crio-a740c7677ba3325728be2b6330f72f5595a4fb0fcd31c360a8228f033a30ecc1 WatchSource:0}: Error finding container a740c7677ba3325728be2b6330f72f5595a4fb0fcd31c360a8228f033a30ecc1: Status 404 returned error can't find the container with id a740c7677ba3325728be2b6330f72f5595a4fb0fcd31c360a8228f033a30ecc1 Nov 03 00:24:54.609318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.609246 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw"] Nov 03 00:24:54.908498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.908419 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" event={"ID":"f9661b07-ed83-4ec4-9948-762b602dff13","Type":"ContainerStarted","Data":"a740c7677ba3325728be2b6330f72f5595a4fb0fcd31c360a8228f033a30ecc1"} Nov 03 00:24:54.909932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.909912 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" Nov 03 00:24:54.910066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.909959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw" event={"ID":"30df45df-f5e5-4ade-818d-d9acede7795a","Type":"ContainerDied","Data":"e8a113ab4ae00298cdfd8e87c2a753d70ab0469ce4ef9765f2048edaf25283dd"} Nov 03 00:24:54.910066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:54.910003 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8a113ab4ae00298cdfd8e87c2a753d70ab0469ce4ef9765f2048edaf25283dd" Nov 03 00:24:55.920193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:55.920159 5144 generic.go:356] "Generic (PLEG): container finished" podID="f9661b07-ed83-4ec4-9948-762b602dff13" containerID="b9a103ba4a3977e56f964f02105c9193410782e3f990c1c061f7caf34dc71497" exitCode=0 Nov 03 00:24:55.920592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:55.920249 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" event={"ID":"f9661b07-ed83-4ec4-9948-762b602dff13","Type":"ContainerDied","Data":"b9a103ba4a3977e56f964f02105c9193410782e3f990c1c061f7caf34dc71497"} Nov 03 00:24:55.996715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:55.996689 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a408be7a-3ba6-4e8f-b030-4cb874baf6fc" path="/var/lib/kubelet/pods/a408be7a-3ba6-4e8f-b030-4cb874baf6fc/volumes" Nov 03 00:24:57.076928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.076905 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:57.168343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.168319 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca\") pod \"f9661b07-ed83-4ec4-9948-762b602dff13\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " Nov 03 00:24:57.168473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.168422 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr\" (UniqueName: \"kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr\") pod \"f9661b07-ed83-4ec4-9948-762b602dff13\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " Nov 03 00:24:57.168542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.168502 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca\") pod \"f9661b07-ed83-4ec4-9948-762b602dff13\" (UID: \"f9661b07-ed83-4ec4-9948-762b602dff13\") " Nov 03 00:24:57.168678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.168652 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca" pod "f9661b07-ed83-4ec4-9948-762b602dff13" (UID: "f9661b07-ed83-4ec4-9948-762b602dff13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:24:57.168848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.168832 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:57.170594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.170563 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca" pod "f9661b07-ed83-4ec4-9948-762b602dff13" (UID: "f9661b07-ed83-4ec4-9948-762b602dff13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:24:57.170769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.170705 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr" pod "f9661b07-ed83-4ec4-9948-762b602dff13" (UID: "f9661b07-ed83-4ec4-9948-762b602dff13"). InnerVolumeSpecName "kube-api-access-2cvcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:24:57.269874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.269837 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9661b07-ed83-4ec4-9948-762b602dff13-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:57.269874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.269863 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cvcr\" (UniqueName: \"kubernetes.io/projected/f9661b07-ed83-4ec4-9948-762b602dff13-kube-api-access-2cvcr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:24:57.938296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.938271 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" Nov 03 00:24:57.938479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.938306 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw" event={"ID":"f9661b07-ed83-4ec4-9948-762b602dff13","Type":"ContainerDied","Data":"a740c7677ba3325728be2b6330f72f5595a4fb0fcd31c360a8228f033a30ecc1"} Nov 03 00:24:57.938479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:57.938404 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a740c7677ba3325728be2b6330f72f5595a4fb0fcd31c360a8228f033a30ecc1" Nov 03 00:24:59.045893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:59.045862 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s"] Nov 03 00:24:59.095949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:59.095923 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-101e0521-q4v4s"] Nov 03 00:24:59.995932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:24:59.995899 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1f144c-5a70-499c-a84a-0e0f0bac5f66" path="/var/lib/kubelet/pods/0a1f144c-5a70-499c-a84a-0e0f0bac5f66/volumes" Nov 03 00:25:00.288146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.288113 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj"] Nov 03 00:25:00.288963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.288939 5144 cpu_manager.go:414] "containerMap: removing container" podUID="30df45df-f5e5-4ade-818d-d9acede7795a" containerName="revoke-keys" Nov 03 00:25:00.289022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.288967 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30df45df-f5e5-4ade-818d-d9acede7795a" containerName="revoke-keys" Nov 03 00:25:00.289022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.289006 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f9661b07-ed83-4ec4-9948-762b602dff13" containerName="revoke-keys" Nov 03 00:25:00.289022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.289013 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9661b07-ed83-4ec4-9948-762b602dff13" containerName="revoke-keys" Nov 03 00:25:00.289195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.289184 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="30df45df-f5e5-4ade-818d-d9acede7795a" containerName="revoke-keys" Nov 03 00:25:00.289229 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.289203 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9661b07-ed83-4ec4-9948-762b602dff13" containerName="revoke-keys" Nov 03 00:25:00.293393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.293378 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.342531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.342504 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj"] Nov 03 00:25:00.386570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.386548 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:25:00.386710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.386596 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:25:00.402143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.402121 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.402218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.402164 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mrfx\" (UniqueName: \"kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.402318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.402302 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.503257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.503228 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.503398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.503381 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.503464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.503435 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mrfx\" (UniqueName: \"kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.504072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.504002 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.505840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.505816 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.561221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.561153 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mrfx\" (UniqueName: \"kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx\") pod \"maas-api-revoke-sub-303bb17f-xq6vj\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.606482 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.606357 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:00.986877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:00.986853 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj"] Nov 03 00:25:00.987317 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:00.987297 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0257d36e_5033_467c_aad4_ce1248609d00.slice/crio-4f6042a28ab7c29ed08414637ef77b459480196c3bf13040e66cfe8805ac4af9 WatchSource:0}: Error finding container 4f6042a28ab7c29ed08414637ef77b459480196c3bf13040e66cfe8805ac4af9: Status 404 returned error can't find the container with id 4f6042a28ab7c29ed08414637ef77b459480196c3bf13040e66cfe8805ac4af9 Nov 03 00:25:01.972317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:01.972290 5144 generic.go:356] "Generic (PLEG): container finished" podID="0257d36e-5033-467c-aad4-ce1248609d00" containerID="193d350b7c375dbbb5326a9c37f8a189406a771c2fbd6be0e5c0a41a363659e2" exitCode=0 Nov 03 00:25:01.972663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:01.972343 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" event={"ID":"0257d36e-5033-467c-aad4-ce1248609d00","Type":"ContainerDied","Data":"193d350b7c375dbbb5326a9c37f8a189406a771c2fbd6be0e5c0a41a363659e2"} Nov 03 00:25:01.972663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:01.972394 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" event={"ID":"0257d36e-5033-467c-aad4-ce1248609d00","Type":"ContainerStarted","Data":"4f6042a28ab7c29ed08414637ef77b459480196c3bf13040e66cfe8805ac4af9"} Nov 03 00:25:03.137506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.137480 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:03.235651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.235567 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca\") pod \"0257d36e-5033-467c-aad4-ce1248609d00\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " Nov 03 00:25:03.235651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.235645 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx\" (UniqueName: \"kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx\") pod \"0257d36e-5033-467c-aad4-ce1248609d00\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " Nov 03 00:25:03.235865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.235729 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca\") pod \"0257d36e-5033-467c-aad4-ce1248609d00\" (UID: \"0257d36e-5033-467c-aad4-ce1248609d00\") " Nov 03 00:25:03.236031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.236007 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca" pod "0257d36e-5033-467c-aad4-ce1248609d00" (UID: "0257d36e-5033-467c-aad4-ce1248609d00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:03.236349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.236325 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:03.238198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.238177 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca" pod "0257d36e-5033-467c-aad4-ce1248609d00" (UID: "0257d36e-5033-467c-aad4-ce1248609d00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:03.238788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.238762 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx" pod "0257d36e-5033-467c-aad4-ce1248609d00" (UID: "0257d36e-5033-467c-aad4-ce1248609d00"). InnerVolumeSpecName "kube-api-access-8mrfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:03.338040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.338013 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0257d36e-5033-467c-aad4-ce1248609d00-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:03.338040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:03.338041 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mrfx\" (UniqueName: \"kubernetes.io/projected/0257d36e-5033-467c-aad4-ce1248609d00-kube-api-access-8mrfx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:04.000935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:04.000908 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" Nov 03 00:25:04.004728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:04.004697 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj" event={"ID":"0257d36e-5033-467c-aad4-ce1248609d00","Type":"ContainerDied","Data":"4f6042a28ab7c29ed08414637ef77b459480196c3bf13040e66cfe8805ac4af9"} Nov 03 00:25:04.004842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:04.004743 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f6042a28ab7c29ed08414637ef77b459480196c3bf13040e66cfe8805ac4af9" Nov 03 00:25:04.049805 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:04.049780 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld"] Nov 03 00:25:04.099619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:04.099594 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60cb67c3-gkzld"] Nov 03 00:25:05.010337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:05.010306 5144 generic.go:356] "Generic (PLEG): container finished" podID="69d3dd78-bb5e-427c-808f-c25ad7e7a650" containerID="b241070880efa4695742574e151849fb463027f6f5f5ebf1705bb4f4fbc9e98d" exitCode=0 Nov 03 00:25:05.010842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:05.010381 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l269t-q5pnz" event={"ID":"69d3dd78-bb5e-427c-808f-c25ad7e7a650","Type":"ContainerDied","Data":"b241070880efa4695742574e151849fb463027f6f5f5ebf1705bb4f4fbc9e98d"} Nov 03 00:25:05.997155 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:05.997117 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d691217e-d678-44b0-a123-60e80a4dcfc3" path="/var/lib/kubelet/pods/d691217e-d678-44b0-a123-60e80a4dcfc3/volumes" Nov 03 00:25:06.203021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.203000 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:25:06.272046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272022 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle\") pod \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " Nov 03 00:25:06.272212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272136 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch\" (UniqueName: \"kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch\") pod \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " Nov 03 00:25:06.272212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272198 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path\") pod \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " Nov 03 00:25:06.272359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272220 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle\") pod \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\" (UID: \"69d3dd78-bb5e-427c-808f-c25ad7e7a650\") " Nov 03 00:25:06.272494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272460 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle" pod "69d3dd78-bb5e-427c-808f-c25ad7e7a650" (UID: "69d3dd78-bb5e-427c-808f-c25ad7e7a650"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:06.272599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272538 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path" pod "69d3dd78-bb5e-427c-808f-c25ad7e7a650" (UID: "69d3dd78-bb5e-427c-808f-c25ad7e7a650"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:06.272708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272690 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle" pod "69d3dd78-bb5e-427c-808f-c25ad7e7a650" (UID: "69d3dd78-bb5e-427c-808f-c25ad7e7a650"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:06.272775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272739 5144 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-service-ca-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.272775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.272752 5144 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/69d3dd78-bb5e-427c-808f-c25ad7e7a650-archives-path\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.274440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.274420 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch" pod "69d3dd78-bb5e-427c-808f-c25ad7e7a650" (UID: "69d3dd78-bb5e-427c-808f-c25ad7e7a650"). InnerVolumeSpecName "kube-api-access-55wch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:06.374440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.374409 5144 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/69d3dd78-bb5e-427c-808f-c25ad7e7a650-trusted-ca-bundle\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:06.374605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:06.374442 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55wch\" (UniqueName: \"kubernetes.io/projected/69d3dd78-bb5e-427c-808f-c25ad7e7a650-kube-api-access-55wch\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:07.027081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.027055 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-l269t-q5pnz" Nov 03 00:25:07.027252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.027084 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-l269t-q5pnz" event={"ID":"69d3dd78-bb5e-427c-808f-c25ad7e7a650","Type":"ContainerDied","Data":"95a6e82028f9c1132516a6dcd7d750a8b7cf4540076bccd34a3b40d376189a93"} Nov 03 00:25:07.027252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.027132 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95a6e82028f9c1132516a6dcd7d750a8b7cf4540076bccd34a3b40d376189a93" Nov 03 00:25:07.643529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.643496 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d"] Nov 03 00:25:07.644163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.644147 5144 cpu_manager.go:414] "containerMap: removing container" podUID="69d3dd78-bb5e-427c-808f-c25ad7e7a650" containerName="insights-gathering" Nov 03 00:25:07.644208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.644165 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69d3dd78-bb5e-427c-808f-c25ad7e7a650" containerName="insights-gathering" Nov 03 00:25:07.644208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.644177 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0257d36e-5033-467c-aad4-ce1248609d00" containerName="revoke-keys" Nov 03 00:25:07.644208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.644181 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0257d36e-5033-467c-aad4-ce1248609d00" containerName="revoke-keys" Nov 03 00:25:07.644306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.644298 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="69d3dd78-bb5e-427c-808f-c25ad7e7a650" containerName="insights-gathering" Nov 03 00:25:07.644337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.644307 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0257d36e-5033-467c-aad4-ce1248609d00" containerName="revoke-keys" Nov 03 00:25:07.648074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.648058 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.709168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.709141 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-cb65f3a5-openshift-default-dockercfg-fxcdx\"" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.820946 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821410 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821497 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821542 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrx7m\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821589 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821676 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821708 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821739 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.822055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.821813 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.832591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.832559 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d"] Nov 03 00:25:07.923223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923116 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923189 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923271 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923320 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrx7m\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923364 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923402 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923433 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923454 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923502 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923862 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.923952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.923927 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.924136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.924109 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.924228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.924195 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.924659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.924586 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.934298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.933580 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.938910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.938870 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.980873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.980847 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:07.981004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:07.980987 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrx7m\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:08.261860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.261835 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:08.691783 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.691708 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d"] Nov 03 00:25:08.730266 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:08.730238 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc275659_89d6_429c_8f7b_65976b66f0c9.slice/crio-c77080e4076146385ddab574e56b8bfe31ae85e0e0460c49b33bd311863b02c1 WatchSource:0}: Error finding container c77080e4076146385ddab574e56b8bfe31ae85e0e0460c49b33bd311863b02c1: Status 404 returned error can't find the container with id c77080e4076146385ddab574e56b8bfe31ae85e0e0460c49b33bd311863b02c1 Nov 03 00:25:08.732813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.732776 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:08.732936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.732864 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:08.732936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.732908 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:08.908427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.908399 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh"] Nov 03 00:25:08.912556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.912533 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:08.970590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:08.970530 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh"] Nov 03 00:25:09.039943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.039914 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.039959 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.039999 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.040051 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.040177 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.040208 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.040267 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.040288 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.040573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.040362 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkmd4\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.049430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.049400 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" event={"ID":"dc275659-89d6-429c-8f7b-65976b66f0c9","Type":"ContainerStarted","Data":"2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677"} Nov 03 00:25:09.049648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.049610 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" event={"ID":"dc275659-89d6-429c-8f7b-65976b66f0c9","Type":"ContainerStarted","Data":"c77080e4076146385ddab574e56b8bfe31ae85e0e0460c49b33bd311863b02c1"} Nov 03 00:25:09.141765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.141731 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.141968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.141946 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142028 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkmd4\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142080 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142115 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142154 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142214 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142412 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142642 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142605 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142692 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.142815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.142734 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.143128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.143108 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.143496 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.143476 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.143689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.143668 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.145114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.145092 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.145413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.145392 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.170668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.170601 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podStartSLOduration=2.170583262 podStartE2EDuration="2.170583262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:09.164143935 +0000 UTC m=+1199.309838391" watchObservedRunningTime="2026-11-03 00:25:09.170583262 +0000 UTC m=+1199.316277721" Nov 03 00:25:09.198290 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.198259 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkmd4\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.198922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.198904 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token\") pod \"e2e-worker-w4-048135-openshift-default-754f957787-lbnjh\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.228211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.228148 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:09.263668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.262952 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:09.265017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.264986 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:25:09.265149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.265042 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 00:25:09.542890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.542858 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx"] Nov 03 00:25:09.548370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.548347 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.613977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.613946 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx"] Nov 03 00:25:09.650005 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.649874 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650005 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.649917 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650005 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.649996 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.650047 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vscbg\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.650107 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.650135 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.650201 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.650283 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.650389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.650307 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.662172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.662149 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh"] Nov 03 00:25:09.663911 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:09.663882 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c4ffff_f066_4253_886b_9de80300737e.slice/crio-f74b1fe184c479c92efd0b4cd31dbce24a1e9765662d3b88c1f7906d9084f01a WatchSource:0}: Error finding container f74b1fe184c479c92efd0b4cd31dbce24a1e9765662d3b88c1f7906d9084f01a: Status 404 returned error can't find the container with id f74b1fe184c479c92efd0b4cd31dbce24a1e9765662d3b88c1f7906d9084f01a Nov 03 00:25:09.666383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.666346 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:09.666460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.666432 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:09.666495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.666479 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:09.751928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.751903 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.751944 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vscbg\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752035 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752081 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752113 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752172 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752209 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752366 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.752943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.752395 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.753276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.753212 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.753586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.753561 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.753727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.753685 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.753833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.753813 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.754049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.754031 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.756405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.756376 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.757253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.757222 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.808942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.808877 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vscbg\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.809222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.809197 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token\") pod \"e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:09.865331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:09.865306 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:10.063222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.063140 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" event={"ID":"91c4ffff-f066-4253-886b-9de80300737e","Type":"ContainerStarted","Data":"ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e"} Nov 03 00:25:10.063222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.063184 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" event={"ID":"91c4ffff-f066-4253-886b-9de80300737e","Type":"ContainerStarted","Data":"f74b1fe184c479c92efd0b4cd31dbce24a1e9765662d3b88c1f7906d9084f01a"} Nov 03 00:25:10.184687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.184613 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podStartSLOduration=2.184597901 podStartE2EDuration="2.184597901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:10.178229845 +0000 UTC m=+1200.323924320" watchObservedRunningTime="2026-11-03 00:25:10.184597901 +0000 UTC m=+1200.330292356" Nov 03 00:25:10.229437 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.229403 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:10.231306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.231280 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:10.231456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.231328 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:10.269893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.269868 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:10.306256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.306228 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:25:10.307139 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.307119 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:25:10.348723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.348698 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx"] Nov 03 00:25:10.358867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.351475 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:10.358867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.351548 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:10.358867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.351588 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:10.550656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:10.544010 5144 scope.go:122] "RemoveContainer" containerID="99355063a2f517f7dfd45cd8d159b9f864282a252fcf21d4f01deaac1316126e" Nov 03 00:25:11.072593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.072541 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" event={"ID":"dbef4fbb-8a67-4440-a29e-f5b41ffd2321","Type":"ContainerStarted","Data":"5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b"} Nov 03 00:25:11.072998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.072662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" event={"ID":"dbef4fbb-8a67-4440-a29e-f5b41ffd2321","Type":"ContainerStarted","Data":"b769c95a95f8f29cb3dbb8f0dc1e45c107fdf9c04a16e7e8b735b84c2d03d080"} Nov 03 00:25:11.073095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.073063 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:11.073768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.073749 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:25:11.199039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.198990 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podStartSLOduration=2.1989765390000002 podStartE2EDuration="2.198976539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:11.193413283 +0000 UTC m=+1201.339107878" watchObservedRunningTime="2026-11-03 00:25:11.198976539 +0000 UTC m=+1201.344670994" Nov 03 00:25:11.232066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.232027 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:11.232378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.232350 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:11.495739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.494360 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr"] Nov 03 00:25:11.495739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.494610 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" containerID="cri-o://0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a" gracePeriod=30 Nov 03 00:25:11.873413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.873381 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:11.873862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.873836 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:11.874053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:11.874029 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:12.197869 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:25:12.197786 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:25:12.243655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:12.237020 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:12.243655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:12.237080 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:12.877658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:12.874494 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:12.877658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:12.874743 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:13.234707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.230996 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:13.234707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.231056 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:13.599588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.599556 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g"] Nov 03 00:25:13.605891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.605871 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.661431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.661394 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.661767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.661747 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.661927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.661910 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.662064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.662049 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6ksc\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.662194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.662181 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.662421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.662405 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.663225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.663204 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.663382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.663363 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.663552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.663537 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.709319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.709294 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g"] Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769373 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769448 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769478 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769516 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6ksc\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769574 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769729 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769764 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769801 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.769890 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.770062 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.770615 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.770903 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.771130 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.773670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.772959 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.774612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.774568 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.775739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.775718 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.877014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.876919 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:13.877014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.876981 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:13.887876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.885819 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.888318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.888294 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6ksc\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:13.936816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:13.936559 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:14.264708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.246146 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:14.268904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.268773 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:14.729651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.728413 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g"] Nov 03 00:25:14.746786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.744592 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:14.753659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.751776 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:14.753659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.751840 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:14.874656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.873664 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:14.874656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:14.873744 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:15.238164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.229383 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:15.238164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.229445 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:15.255708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.254791 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99"] Nov 03 00:25:15.436448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.431904 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ecd699c-g2d99"] Nov 03 00:25:15.520706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.516699 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" event={"ID":"d418e880-f0d2-466a-8761-a2f1328d587e","Type":"ContainerStarted","Data":"2afe78bcbd3d816a803e4fd8abcae9d112a7b28a965083e177189c7ef0cf8770"} Nov 03 00:25:15.520706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.516814 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" event={"ID":"d418e880-f0d2-466a-8761-a2f1328d587e","Type":"ContainerStarted","Data":"e8325ca6d9dc807f84b4296b0a745aa57f25e93962d765401d514c8a3c47b3e2"} Nov 03 00:25:15.671659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.668405 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded" Nov 03 00:25:15.671659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.670498 5144 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/health\": context deadline exceeded" Nov 03 00:25:15.690935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.690045 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podStartSLOduration=2.6900253039999997 podStartE2EDuration="2.690025304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:15.668337851 +0000 UTC m=+1205.814032314" watchObservedRunningTime="2026-11-03 00:25:15.690025304 +0000 UTC m=+1205.835719761" Nov 03 00:25:15.810864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.809704 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:15.866100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.865684 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:15.866100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.865738 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:15.880301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.878741 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:15.880301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.879673 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:15.946954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:15.945766 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:16.030661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.018493 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:16.055877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.055843 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed5b01a-b737-4afc-b2b2-d3d8ed008559" path="/var/lib/kubelet/pods/fed5b01a-b737-4afc-b2b2-d3d8ed008559/volumes" Nov 03 00:25:16.093874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.093720 5144 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": context deadline exceeded" Nov 03 00:25:16.246656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.232173 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:16.246961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.246920 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:16.674153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.674080 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:16.676407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.676382 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:25:16.857720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.856396 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": context deadline exceeded" Nov 03 00:25:16.857720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.857419 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:25:16.910656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.870919 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:16.912012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:16.911940 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:17.041199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.038575 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.041199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.038658 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" Nov 03 00:25:17.041199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.039580 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:17.119132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.089434 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 00:25:17.157789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.133905 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.157789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.133966 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:17.294574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.294281 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 00:25:17.319669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.319588 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.367969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.367706 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" Nov 03 00:25:17.482577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.482550 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:25:17.493208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493182 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493239 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493266 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493336 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493358 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493400 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493483 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493524 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.493713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.493700 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo\") pod \"a2174356-8e04-4e28-9d10-a6041459d108\" (UID: \"a2174356-8e04-4e28-9d10-a6041459d108\") " Nov 03 00:25:17.494533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.494507 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:17.495006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.494960 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:17.495257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.495231 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:17.495343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.495242 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:17.495526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.495507 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:17.497107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.497082 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:25:17.497231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.497152 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:25:17.498103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.498064 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "kube-api-access-f85sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:17.498623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.498599 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token" pod "a2174356-8e04-4e28-9d10-a6041459d108" (UID: "a2174356-8e04-4e28-9d10-a6041459d108"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595741 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595775 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595793 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595807 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595822 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595835 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2174356-8e04-4e28-9d10-a6041459d108-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595849 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85sr\" (UniqueName: \"kubernetes.io/projected/a2174356-8e04-4e28-9d10-a6041459d108-kube-api-access-f85sr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595868 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2174356-8e04-4e28-9d10-a6041459d108-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.595915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.595883 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2174356-8e04-4e28-9d10-a6041459d108-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:17.685412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.685315 5144 generic.go:356] "Generic (PLEG): container finished" podID="a2174356-8e04-4e28-9d10-a6041459d108" containerID="0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a" exitCode=0 Nov 03 00:25:17.687018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.686990 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" event={"ID":"a2174356-8e04-4e28-9d10-a6041459d108","Type":"ContainerDied","Data":"0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a"} Nov 03 00:25:17.687421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.687399 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" event={"ID":"a2174356-8e04-4e28-9d10-a6041459d108","Type":"ContainerDied","Data":"3148211c2ec0bda98649b2815b39d0cf0c9519789653d691087966e6c43f628b"} Nov 03 00:25:17.687676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.687120 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr" Nov 03 00:25:17.688100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.687822 5144 scope.go:122] "RemoveContainer" containerID="0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a" Nov 03 00:25:17.703478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.703282 5144 scope.go:122] "RemoveContainer" containerID="0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a" Nov 03 00:25:17.703559 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:25:17.703537 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a\": container with ID starting with 0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a not found: ID does not exist" containerID="0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a" Nov 03 00:25:17.703607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.703566 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a"} err="failed to get container status \"0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a\": rpc error: code = NotFound desc = could not find container \"0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a\": container with ID starting with 0fe0f421cdf7130acf7340c9ceda0281ef35470d7d7898f78c6f2245bdd5299a not found: ID does not exist" Nov 03 00:25:17.871719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.867516 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.871911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.871759 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": dial tcp 10.132.0.126:15021: connect: connection refused" Nov 03 00:25:17.887496 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.887465 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr"] Nov 03 00:25:17.889285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.889247 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-ttwgr"] Nov 03 00:25:17.942167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.941982 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:17.942167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.942039 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:17.998158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:17.998127 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2174356-8e04-4e28-9d10-a6041459d108" path="/var/lib/kubelet/pods/a2174356-8e04-4e28-9d10-a6041459d108/volumes" Nov 03 00:25:18.535047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:18.535008 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:18.537054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:18.537033 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:18.547653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:18.543863 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:25:18.949749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:18.948000 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:18.949749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:18.948072 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:19.866689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:19.866647 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:25:19.866920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:19.866720 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:19.945656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:19.939917 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:19.945656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:19.939968 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:20.456658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.456129 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2"] Nov 03 00:25:20.463052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.463023 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" Nov 03 00:25:20.463052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.463049 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" Nov 03 00:25:20.463250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.463237 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2174356-8e04-4e28-9d10-a6041459d108" containerName="istio-proxy" Nov 03 00:25:20.468670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.468614 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.498336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.498311 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2"] Nov 03 00:25:20.545374 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.544423 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:25:20.545374 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.545075 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:25:20.579184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.579150 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.579411 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.579351 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb7th\" (UniqueName: \"kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.579472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.579426 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.611562 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.611535 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:20.612014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.611996 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:20.612666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.612616 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:25:20.680288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.680250 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.680451 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.680365 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb7th\" (UniqueName: \"kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.680530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.680464 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.681345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.681318 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.683128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.683104 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.770693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.763136 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb7th\" (UniqueName: \"kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th\") pod \"maas-api-revoke-sub-000e56f7-z44x2\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.784314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.784286 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:20.955251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.955201 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:20.955446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:20.955284 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:21.208976 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:21.208946 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b184a0_3764_4cb3_bdde_71cb8e9d5fc3.slice/crio-bf3c0e07e6ebff7d6ac7d11896a694970357a1963b09709f6fef44d1050ba4f6 WatchSource:0}: Error finding container bf3c0e07e6ebff7d6ac7d11896a694970357a1963b09709f6fef44d1050ba4f6: Status 404 returned error can't find the container with id bf3c0e07e6ebff7d6ac7d11896a694970357a1963b09709f6fef44d1050ba4f6 Nov 03 00:25:21.209447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.209423 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2"] Nov 03 00:25:21.380329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.380299 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2"] Nov 03 00:25:21.386604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.386577 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.472411 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.467713 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2"] Nov 03 00:25:21.495958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.495923 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.495968 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496079 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5prh\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496305 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496350 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496382 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496407 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496493 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.496785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.496573 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598097 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598141 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598167 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598189 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598262 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598331 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598372 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598402 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598470 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598514 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598546 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5prh\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.598806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.598591 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.599101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.599020 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.599101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.599151 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.600599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.600579 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.600864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.600847 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.654854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.654827 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.655409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.655382 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5prh\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh\") pod \"e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.701450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.701424 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:21.777449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.777411 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" event={"ID":"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3","Type":"ContainerStarted","Data":"9479e4366b699bd9c5ee220a90205d8988c8805133e41a1cac6e05afe6644a27"} Nov 03 00:25:21.777662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.777472 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" event={"ID":"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3","Type":"ContainerStarted","Data":"bf3c0e07e6ebff7d6ac7d11896a694970357a1963b09709f6fef44d1050ba4f6"} Nov 03 00:25:21.896575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.896520 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" podStartSLOduration=1.896506512 podStartE2EDuration="1.896506512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:21.892498802 +0000 UTC m=+1212.038193276" watchObservedRunningTime="2026-11-03 00:25:21.896506512 +0000 UTC m=+1212.042200967" Nov 03 00:25:21.940659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.914150 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6"] Nov 03 00:25:21.956734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.956708 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:21.956937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.956914 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:21.979043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.979015 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:21.983147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:21.982846 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6"] Nov 03 00:25:22.004384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.004360 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7tz9\" (UniqueName: \"kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.004515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.004414 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.004515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.004453 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.105614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.105522 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.105801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.105616 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.105913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.105896 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7tz9\" (UniqueName: \"kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.106346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.106266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.108079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.108057 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.162458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.162423 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7tz9\" (UniqueName: \"kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9\") pod \"maas-api-revoke-sub-f532143a-pt5t6\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.245663 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:22.245609 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36e621a1_9466_4ae0_a014_38621b0175de.slice/crio-b23d05c8b0f715e82b135414179a3d385b2ae65e7dfa945b2ff820f23b30c863 WatchSource:0}: Error finding container b23d05c8b0f715e82b135414179a3d385b2ae65e7dfa945b2ff820f23b30c863: Status 404 returned error can't find the container with id b23d05c8b0f715e82b135414179a3d385b2ae65e7dfa945b2ff820f23b30c863 Nov 03 00:25:22.248650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.248596 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:22.248762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.248741 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:22.248828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.248812 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:22.249413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.249393 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2"] Nov 03 00:25:22.294653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.294617 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:22.695842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.695737 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6"] Nov 03 00:25:22.704030 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:22.704002 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda330dd9f_7cb9_4e46_83cb_1dc9f16a1d60.slice/crio-a759d285dd09d9d7d64eca247281f10f7c94a0ba9d17f57bfbfbf5e6338aa82f WatchSource:0}: Error finding container a759d285dd09d9d7d64eca247281f10f7c94a0ba9d17f57bfbfbf5e6338aa82f: Status 404 returned error can't find the container with id a759d285dd09d9d7d64eca247281f10f7c94a0ba9d17f57bfbfbf5e6338aa82f Nov 03 00:25:22.791423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.791386 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" event={"ID":"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60","Type":"ContainerStarted","Data":"a759d285dd09d9d7d64eca247281f10f7c94a0ba9d17f57bfbfbf5e6338aa82f"} Nov 03 00:25:22.793748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.793721 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" event={"ID":"36e621a1-9466-4ae0-a014-38621b0175de","Type":"ContainerStarted","Data":"dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5"} Nov 03 00:25:22.793894 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.793773 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" event={"ID":"36e621a1-9466-4ae0-a014-38621b0175de","Type":"ContainerStarted","Data":"b23d05c8b0f715e82b135414179a3d385b2ae65e7dfa945b2ff820f23b30c863"} Nov 03 00:25:22.795730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.795703 5144 generic.go:356] "Generic (PLEG): container finished" podID="f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" containerID="9479e4366b699bd9c5ee220a90205d8988c8805133e41a1cac6e05afe6644a27" exitCode=0 Nov 03 00:25:22.795828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.795750 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" event={"ID":"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3","Type":"ContainerDied","Data":"9479e4366b699bd9c5ee220a90205d8988c8805133e41a1cac6e05afe6644a27"} Nov 03 00:25:22.949272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.949135 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podStartSLOduration=1.94911279 podStartE2EDuration="1.94911279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:22.909686071 +0000 UTC m=+1213.055380547" watchObservedRunningTime="2026-11-03 00:25:22.94911279 +0000 UTC m=+1213.094807252" Nov 03 00:25:22.955546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.955516 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" start-of-body= Nov 03 00:25:22.955813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:22.955785 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.127:15021/healthz/ready\": dial tcp 10.132.0.127:15021: connect: connection refused" Nov 03 00:25:23.701792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.701757 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:23.703290 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.703260 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:25:23.703426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.703318 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 00:25:23.805857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.805823 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" event={"ID":"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60","Type":"ContainerStarted","Data":"e822ffb7e1618672fb24c309bdcf7443743f26fbc0a7b6e1566941712dafa2ab"} Nov 03 00:25:23.946075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.946016 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" podStartSLOduration=2.945998831 podStartE2EDuration="2.945998831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:23.941058225 +0000 UTC m=+1214.086752698" watchObservedRunningTime="2026-11-03 00:25:23.945998831 +0000 UTC m=+1214.091693288" Nov 03 00:25:23.948925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.948897 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:23.949494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.949418 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:23.950244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:23.950224 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:25:24.047094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.047070 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:24.176331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.176300 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca\") pod \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " Nov 03 00:25:24.176523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.176418 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca\") pod \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " Nov 03 00:25:24.176593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.176529 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th\" (UniqueName: \"kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th\") pod \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\" (UID: \"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3\") " Nov 03 00:25:24.176713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.176686 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca" pod "f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" (UID: "f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:24.177154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.177128 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:24.178565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.178543 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca" pod "f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" (UID: "f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:24.178785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.178623 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th" pod "f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" (UID: "f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3"). InnerVolumeSpecName "kube-api-access-rb7th". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:24.278906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.278876 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:24.278906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.278902 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb7th\" (UniqueName: \"kubernetes.io/projected/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3-kube-api-access-rb7th\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:24.702022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.701940 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:25:24.702022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.702001 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 00:25:24.823652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.819794 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" event={"ID":"f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3","Type":"ContainerDied","Data":"bf3c0e07e6ebff7d6ac7d11896a694970357a1963b09709f6fef44d1050ba4f6"} Nov 03 00:25:24.823652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.819817 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2" Nov 03 00:25:24.823652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.819857 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf3c0e07e6ebff7d6ac7d11896a694970357a1963b09709f6fef44d1050ba4f6" Nov 03 00:25:24.846294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.846260 5144 generic.go:356] "Generic (PLEG): container finished" podID="a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" containerID="e822ffb7e1618672fb24c309bdcf7443743f26fbc0a7b6e1566941712dafa2ab" exitCode=0 Nov 03 00:25:24.846770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:24.846740 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" event={"ID":"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60","Type":"ContainerDied","Data":"e822ffb7e1618672fb24c309bdcf7443743f26fbc0a7b6e1566941712dafa2ab"} Nov 03 00:25:25.702477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:25.702445 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:25:25.702906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:25.702506 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 00:25:26.089930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.089901 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:26.209925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.209892 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9\" (UniqueName: \"kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9\") pod \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " Nov 03 00:25:26.210122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.209945 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca\") pod \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " Nov 03 00:25:26.210122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.209994 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca\") pod \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\" (UID: \"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60\") " Nov 03 00:25:26.210675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.210642 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca" pod "a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" (UID: "a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:26.212625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.212600 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9" pod "a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" (UID: "a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60"). InnerVolumeSpecName "kube-api-access-k7tz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:26.212856 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.212665 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca" pod "a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" (UID: "a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:26.311645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.311558 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7tz9\" (UniqueName: \"kubernetes.io/projected/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-kube-api-access-k7tz9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:26.311645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.311590 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:26.311645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.311600 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:26.702854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.702779 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:25:26.702854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.702830 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 00:25:26.865192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.865160 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" Nov 03 00:25:26.865382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.865193 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6" event={"ID":"a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60","Type":"ContainerDied","Data":"a759d285dd09d9d7d64eca247281f10f7c94a0ba9d17f57bfbfbf5e6338aa82f"} Nov 03 00:25:26.865382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.865246 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a759d285dd09d9d7d64eca247281f10f7c94a0ba9d17f57bfbfbf5e6338aa82f" Nov 03 00:25:26.950131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950100 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5"] Nov 03 00:25:26.950745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950730 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" containerName="revoke-keys" Nov 03 00:25:26.950816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950746 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" containerName="revoke-keys" Nov 03 00:25:26.950816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950767 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" containerName="revoke-keys" Nov 03 00:25:26.950816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950772 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" containerName="revoke-keys" Nov 03 00:25:26.950910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950882 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" containerName="revoke-keys" Nov 03 00:25:26.950910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.950892 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" containerName="revoke-keys" Nov 03 00:25:26.954454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:26.954405 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.006295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.006267 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:25:27.006474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.006405 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:25:27.011173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.010571 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5"] Nov 03 00:25:27.046406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.046375 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.046556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.046445 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.046686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.046586 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9rph\" (UniqueName: \"kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.147808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.147769 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.148014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.147829 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.148014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.147923 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9rph\" (UniqueName: \"kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.148519 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.148498 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.150434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.150410 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.205409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.205352 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9rph\" (UniqueName: \"kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph\") pod \"maas-api-revoke-sub-21ae6c16-h5hq5\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.271119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.271089 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:27.676366 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:27.676336 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f4dfb68_7089_48f5_9253_5a833be259e2.slice/crio-821019e23a4e327be1f1cdef11f9438340f3d6e110c8d3cc9d676ac61a64b7e1 WatchSource:0}: Error finding container 821019e23a4e327be1f1cdef11f9438340f3d6e110c8d3cc9d676ac61a64b7e1: Status 404 returned error can't find the container with id 821019e23a4e327be1f1cdef11f9438340f3d6e110c8d3cc9d676ac61a64b7e1 Nov 03 00:25:27.676588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.676565 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5"] Nov 03 00:25:27.702586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.702559 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:25:27.702711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.702614 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 00:25:27.875290 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:27.875257 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" event={"ID":"3f4dfb68-7089-48f5-9253-5a833be259e2","Type":"ContainerStarted","Data":"821019e23a4e327be1f1cdef11f9438340f3d6e110c8d3cc9d676ac61a64b7e1"} Nov 03 00:25:28.702625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:28.702591 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 00:25:28.702814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:28.702654 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 00:25:28.886892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:28.886859 5144 generic.go:356] "Generic (PLEG): container finished" podID="3f4dfb68-7089-48f5-9253-5a833be259e2" containerID="a98e65dee02a0e82cefe2fb6a90b95ac338815ae4d1e6b468368c3887d1d52e7" exitCode=0 Nov 03 00:25:28.887277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:28.886897 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" event={"ID":"3f4dfb68-7089-48f5-9253-5a833be259e2","Type":"ContainerDied","Data":"a98e65dee02a0e82cefe2fb6a90b95ac338815ae4d1e6b468368c3887d1d52e7"} Nov 03 00:25:29.707158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:29.707131 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:29.707658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:29.707625 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:29.708183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:29.708168 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:25:30.104249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.104224 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:30.193549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.193521 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca\") pod \"3f4dfb68-7089-48f5-9253-5a833be259e2\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " Nov 03 00:25:30.193743 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.193624 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca\") pod \"3f4dfb68-7089-48f5-9253-5a833be259e2\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " Nov 03 00:25:30.193743 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.193673 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph\" (UniqueName: \"kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph\") pod \"3f4dfb68-7089-48f5-9253-5a833be259e2\" (UID: \"3f4dfb68-7089-48f5-9253-5a833be259e2\") " Nov 03 00:25:30.194073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.194048 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca" pod "3f4dfb68-7089-48f5-9253-5a833be259e2" (UID: "3f4dfb68-7089-48f5-9253-5a833be259e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:30.194375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.194356 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:30.195836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.195808 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca" pod "3f4dfb68-7089-48f5-9253-5a833be259e2" (UID: "3f4dfb68-7089-48f5-9253-5a833be259e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:30.195941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.195886 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph" pod "3f4dfb68-7089-48f5-9253-5a833be259e2" (UID: "3f4dfb68-7089-48f5-9253-5a833be259e2"). InnerVolumeSpecName "kube-api-access-f9rph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:30.295892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.295863 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9rph\" (UniqueName: \"kubernetes.io/projected/3f4dfb68-7089-48f5-9253-5a833be259e2-kube-api-access-f9rph\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:30.295892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.295894 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f4dfb68-7089-48f5-9253-5a833be259e2-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:30.904257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.904226 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" Nov 03 00:25:30.904427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.904270 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5" event={"ID":"3f4dfb68-7089-48f5-9253-5a833be259e2","Type":"ContainerDied","Data":"821019e23a4e327be1f1cdef11f9438340f3d6e110c8d3cc9d676ac61a64b7e1"} Nov 03 00:25:30.904427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:30.904315 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="821019e23a4e327be1f1cdef11f9438340f3d6e110c8d3cc9d676ac61a64b7e1" Nov 03 00:25:32.041675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.041626 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz"] Nov 03 00:25:32.042279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.042266 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3f4dfb68-7089-48f5-9253-5a833be259e2" containerName="revoke-keys" Nov 03 00:25:32.042334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.042280 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f4dfb68-7089-48f5-9253-5a833be259e2" containerName="revoke-keys" Nov 03 00:25:32.042422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.042411 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f4dfb68-7089-48f5-9253-5a833be259e2" containerName="revoke-keys" Nov 03 00:25:32.045983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.045968 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118161 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118219 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118297 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118405 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118438 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118496 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x56sv\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118585 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118615 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.122934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.118683 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.187389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.187351 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a9404c-openshift-default-dockercfg-sv2nl\"" Nov 03 00:25:32.198941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.198848 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz"] Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221091 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221148 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221207 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221261 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221307 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221357 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221469 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221509 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.221572 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x56sv\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.222163 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.222327 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.223151 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.224654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.224484 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.245592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.245530 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.246313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.246288 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.273572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.271257 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.279182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.279151 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.280881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.280857 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x56sv\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv\") pod \"e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.359580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.359489 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:32.821441 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:32.821410 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3795cc1f_3ad6_4676_a9b9_e36061737693.slice/crio-dd27932f2e919d4a4f4a9d2033c26561270ac8b313d402aea3c49dbcd0ab2115 WatchSource:0}: Error finding container dd27932f2e919d4a4f4a9d2033c26561270ac8b313d402aea3c49dbcd0ab2115: Status 404 returned error can't find the container with id dd27932f2e919d4a4f4a9d2033c26561270ac8b313d402aea3c49dbcd0ab2115 Nov 03 00:25:32.821703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.821687 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz"] Nov 03 00:25:32.824433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.824402 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:32.824490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.824466 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:32.824525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.824514 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:25:32.925094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.925062 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" event={"ID":"3795cc1f-3ad6-4676-a9b9-e36061737693","Type":"ContainerStarted","Data":"845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51"} Nov 03 00:25:32.925235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:32.925109 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" event={"ID":"3795cc1f-3ad6-4676-a9b9-e36061737693","Type":"ContainerStarted","Data":"dd27932f2e919d4a4f4a9d2033c26561270ac8b313d402aea3c49dbcd0ab2115"} Nov 03 00:25:33.051468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:33.051410 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podStartSLOduration=2.05139252 podStartE2EDuration="2.05139252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:33.047287771 +0000 UTC m=+1223.192982244" watchObservedRunningTime="2026-11-03 00:25:33.05139252 +0000 UTC m=+1223.197086979" Nov 03 00:25:33.360491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:33.360464 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:33.365301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:33.365277 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:33.933998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:33.933968 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:33.935362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:33.935339 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:25:35.894744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.894713 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-t69zx"] Nov 03 00:25:35.899732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.899712 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:35.956315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.956291 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-t69zx"] Nov 03 00:25:35.979075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.979046 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:35.979217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.979122 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:35.979217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.979158 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:35.979380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.979358 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gk95\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:35.999119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.999091 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 00:25:35.999119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.999091 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-0fb7528a8dea\"" Nov 03 00:25:35.999325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.999157 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 00:25:35.999325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:35.999159 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pnghk\"" Nov 03 00:25:36.003925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.003907 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 00:25:36.003925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.003915 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 00:25:36.080179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.080151 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gk95\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.080411 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.080392 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.080484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.080467 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.080569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.080527 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.081294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.081273 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.081361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.081316 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.082908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.082890 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.139671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.139606 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gk95\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95\") pod \"payload-processing-external-model-698cf49b84-t69zx\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.213709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.213621 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:36.227692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.227669 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b"] Nov 03 00:25:36.233278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.233259 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.283825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.283735 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.283920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.283822 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tpvm\" (UniqueName: \"kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.283920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.283899 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.298676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.287875 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:25:36.298676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.287959 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:25:36.298676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.288340 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b"] Nov 03 00:25:36.385413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.385386 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.385551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.385454 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tpvm\" (UniqueName: \"kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.385551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.385513 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.386200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.386174 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.388687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.388648 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.470865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.470765 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tpvm\" (UniqueName: \"kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm\") pod \"maas-api-revoke-sub-65481551-m6r8b\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.566354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.566317 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:36.651045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.651002 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-t69zx"] Nov 03 00:25:36.957120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.957094 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b"] Nov 03 00:25:36.958675 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:36.958645 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff6e014_d317_47f5_a224_bc02ece6a9c4.slice/crio-e791c6d567ae1d39be17d97f68cd33d9bb3eb1f858ad0057e9ad79825f7663a9 WatchSource:0}: Error finding container e791c6d567ae1d39be17d97f68cd33d9bb3eb1f858ad0057e9ad79825f7663a9: Status 404 returned error can't find the container with id e791c6d567ae1d39be17d97f68cd33d9bb3eb1f858ad0057e9ad79825f7663a9 Nov 03 00:25:36.969483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.969458 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" event={"ID":"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3","Type":"ContainerStarted","Data":"68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9"} Nov 03 00:25:36.969596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.969511 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" event={"ID":"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3","Type":"ContainerStarted","Data":"2552955b91ec2f06e1cacf34b110916c00853641b3fbccb81098993d0df2da7c"} Nov 03 00:25:36.969596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:36.969561 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:37.081969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:37.081921 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" podStartSLOduration=2.081908166 podStartE2EDuration="2.081908166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:37.080938899 +0000 UTC m=+1227.226633356" watchObservedRunningTime="2026-11-03 00:25:37.081908166 +0000 UTC m=+1227.227602648" Nov 03 00:25:37.979228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:37.979148 5144 generic.go:356] "Generic (PLEG): container finished" podID="dff6e014-d317-47f5-a224-bc02ece6a9c4" containerID="bc043dbf7aefb6c58a40bd6cdeb189c802aeba59360e739e1f2e53506282bfd4" exitCode=0 Nov 03 00:25:37.979228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:37.979217 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" event={"ID":"dff6e014-d317-47f5-a224-bc02ece6a9c4","Type":"ContainerDied","Data":"bc043dbf7aefb6c58a40bd6cdeb189c802aeba59360e739e1f2e53506282bfd4"} Nov 03 00:25:37.979671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:37.979266 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" event={"ID":"dff6e014-d317-47f5-a224-bc02ece6a9c4","Type":"ContainerStarted","Data":"e791c6d567ae1d39be17d97f68cd33d9bb3eb1f858ad0057e9ad79825f7663a9"} Nov 03 00:25:39.154250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.154220 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:39.229982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.229958 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca\") pod \"dff6e014-d317-47f5-a224-bc02ece6a9c4\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " Nov 03 00:25:39.230182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.230017 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca\") pod \"dff6e014-d317-47f5-a224-bc02ece6a9c4\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " Nov 03 00:25:39.230182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.230168 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm\" (UniqueName: \"kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm\") pod \"dff6e014-d317-47f5-a224-bc02ece6a9c4\" (UID: \"dff6e014-d317-47f5-a224-bc02ece6a9c4\") " Nov 03 00:25:39.230379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.230356 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca" pod "dff6e014-d317-47f5-a224-bc02ece6a9c4" (UID: "dff6e014-d317-47f5-a224-bc02ece6a9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:39.230543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.230526 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:39.232294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.232270 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm" pod "dff6e014-d317-47f5-a224-bc02ece6a9c4" (UID: "dff6e014-d317-47f5-a224-bc02ece6a9c4"). InnerVolumeSpecName "kube-api-access-4tpvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:39.232407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.232315 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca" pod "dff6e014-d317-47f5-a224-bc02ece6a9c4" (UID: "dff6e014-d317-47f5-a224-bc02ece6a9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:25:39.332304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.332277 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tpvm\" (UniqueName: \"kubernetes.io/projected/dff6e014-d317-47f5-a224-bc02ece6a9c4-kube-api-access-4tpvm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:39.332304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:39.332302 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dff6e014-d317-47f5-a224-bc02ece6a9c4-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:40.077656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:40.069854 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" event={"ID":"dff6e014-d317-47f5-a224-bc02ece6a9c4","Type":"ContainerDied","Data":"e791c6d567ae1d39be17d97f68cd33d9bb3eb1f858ad0057e9ad79825f7663a9"} Nov 03 00:25:40.077656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:40.069924 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e791c6d567ae1d39be17d97f68cd33d9bb3eb1f858ad0057e9ad79825f7663a9" Nov 03 00:25:40.077656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:40.070002 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b" Nov 03 00:25:41.367910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:41.367856 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:41.868655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:41.867735 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:41.898694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:41.898616 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.84:8000/ready\": context deadline exceeded" Nov 03 00:25:42.011520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:42.011483 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 00:25:42.289857 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:25:42.289814 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:25:44.760419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.760381 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft"] Nov 03 00:25:44.761495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.761474 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dff6e014-d317-47f5-a224-bc02ece6a9c4" containerName="revoke-keys" Nov 03 00:25:44.761541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.761499 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dff6e014-d317-47f5-a224-bc02ece6a9c4" containerName="revoke-keys" Nov 03 00:25:44.761727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.761712 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dff6e014-d317-47f5-a224-bc02ece6a9c4" containerName="revoke-keys" Nov 03 00:25:44.767053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.767036 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:44.821051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.821027 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft"] Nov 03 00:25:44.844153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.842892 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgfv5\" (UniqueName: \"kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5\") pod \"payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:44.844153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.843965 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:44.876747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.872731 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a9404c-dockercfg-lbxc7\"" Nov 03 00:25:44.876747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.872822 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a9404c\"" Nov 03 00:25:44.956516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.953556 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgfv5\" (UniqueName: \"kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5\") pod \"payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:44.956516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.953761 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:44.956516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:44.954777 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:45.002772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.002739 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks"] Nov 03 00:25:45.015727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.015661 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.033788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.033750 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgfv5\" (UniqueName: \"kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5\") pod \"payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:45.048801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.048775 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks"] Nov 03 00:25:45.081192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.081166 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:45.164009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.162600 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9404c-79684694cc-wbnks\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.164009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.163044 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmvs6\" (UniqueName: \"kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6\") pod \"payload-processing-e2e-api-a-a9404c-79684694cc-wbnks\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.264824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.264797 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9404c-79684694cc-wbnks\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.265009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.264994 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmvs6\" (UniqueName: \"kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6\") pod \"payload-processing-e2e-api-a-a9404c-79684694cc-wbnks\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.265763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.265718 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a9404c-79684694cc-wbnks\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.321300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.321271 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmvs6\" (UniqueName: \"kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6\") pod \"payload-processing-e2e-api-a-a9404c-79684694cc-wbnks\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.351647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.351612 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:45.477135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.477107 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft"] Nov 03 00:25:45.481496 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:45.481464 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bef802_1658_472f_9df7_3a8d88bf6c07.slice/crio-eccfcbf5bb76dda76554ce2373c19c3342dfd062beef7a657fdc3a1c8a5b2af5 WatchSource:0}: Error finding container eccfcbf5bb76dda76554ce2373c19c3342dfd062beef7a657fdc3a1c8a5b2af5: Status 404 returned error can't find the container with id eccfcbf5bb76dda76554ce2373c19c3342dfd062beef7a657fdc3a1c8a5b2af5 Nov 03 00:25:45.730400 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:45.730372 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c9bbfd_1370_4771_8db1_3cac8c9a93ce.slice/crio-e2535d2d966cbbb9532279d96a752617685ada7d7e734cf87238d9d1b8ca6b3a WatchSource:0}: Error finding container e2535d2d966cbbb9532279d96a752617685ada7d7e734cf87238d9d1b8ca6b3a: Status 404 returned error can't find the container with id e2535d2d966cbbb9532279d96a752617685ada7d7e734cf87238d9d1b8ca6b3a Nov 03 00:25:45.731903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:45.731876 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks"] Nov 03 00:25:46.271332 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.271289 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" event={"ID":"64c9bbfd-1370-4771-8db1-3cac8c9a93ce","Type":"ContainerStarted","Data":"3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c"} Nov 03 00:25:46.271823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.271394 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" event={"ID":"64c9bbfd-1370-4771-8db1-3cac8c9a93ce","Type":"ContainerStarted","Data":"e2535d2d966cbbb9532279d96a752617685ada7d7e734cf87238d9d1b8ca6b3a"} Nov 03 00:25:46.271823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.271557 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:46.273302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.273274 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" event={"ID":"c5bef802-1658-472f-9df7-3a8d88bf6c07","Type":"ContainerStarted","Data":"2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57"} Nov 03 00:25:46.273424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.273311 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" event={"ID":"c5bef802-1658-472f-9df7-3a8d88bf6c07","Type":"ContainerStarted","Data":"eccfcbf5bb76dda76554ce2373c19c3342dfd062beef7a657fdc3a1c8a5b2af5"} Nov 03 00:25:46.273424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.273410 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:46.390486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.390440 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" podStartSLOduration=2.390427378 podStartE2EDuration="2.390427378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:46.388827514 +0000 UTC m=+1236.534521990" watchObservedRunningTime="2026-11-03 00:25:46.390427378 +0000 UTC m=+1236.536121858" Nov 03 00:25:46.505257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:46.505212 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" podStartSLOduration=2.505199137 podStartE2EDuration="2.505199137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:46.502856982 +0000 UTC m=+1236.648551459" watchObservedRunningTime="2026-11-03 00:25:46.505199137 +0000 UTC m=+1236.650893592" Nov 03 00:25:47.982526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:47.982488 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:25:50.362435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:50.362404 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-t69zx"] Nov 03 00:25:50.363731 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:50.363704 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:25:54.909180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:54.909069 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n"] Nov 03 00:25:54.933607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:54.933575 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:54.950055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:54.950030 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n"] Nov 03 00:25:54.994817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:54.994789 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-73821c\"" Nov 03 00:25:54.995117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:54.994807 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-73821c-dockercfg-dm7wr\"" Nov 03 00:25:55.004432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.004395 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg96t\" (UniqueName: \"kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t\") pod \"payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.004651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.004615 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.093333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.093295 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw"] Nov 03 00:25:55.100680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.100652 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.113665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.108908 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.113665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.109030 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg96t\" (UniqueName: \"kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t\") pod \"payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.113665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.110512 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.140942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.140910 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw"] Nov 03 00:25:55.166488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.166404 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg96t\" (UniqueName: \"kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t\") pod \"payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.210715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.210687 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.210864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.210805 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fq4g\" (UniqueName: \"kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g\") pod \"payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.248510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.248483 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:55.312582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.312552 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.312758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.312672 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fq4g\" (UniqueName: \"kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g\") pod \"payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.315327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.315299 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.363074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.363033 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerName="payload-processing" containerID="cri-o://68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9" gracePeriod=25 Nov 03 00:25:55.370850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.370818 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fq4g\" (UniqueName: \"kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g\") pod \"payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.446962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.446897 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:55.643978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.643814 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n"] Nov 03 00:25:55.687087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.687057 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:55.824350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.824315 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95\") pod \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " Nov 03 00:25:55.824526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.824411 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume\") pod \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " Nov 03 00:25:55.824526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.824461 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials\") pod \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " Nov 03 00:25:55.824675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.824657 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume\") pod \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\" (UID: \"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3\") " Nov 03 00:25:55.825021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.824995 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume" pod "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" (UID: "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:55.825359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.825267 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:55.825543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.825516 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume" pod "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" (UID: "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:25:55.827164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.827128 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95" pod "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" (UID: "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3"). InnerVolumeSpecName "kube-api-access-6gk95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:55.829117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.829097 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials" pod "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" (UID: "ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:25:55.832118 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:25:55.832093 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2621f2a0_7c0a_42cd_b3ba_2bf8493a78d4.slice/crio-6f90f5e29907936e1e77544615459208b7796e502fd7ccae11f347b2111ef777 WatchSource:0}: Error finding container 6f90f5e29907936e1e77544615459208b7796e502fd7ccae11f347b2111ef777: Status 404 returned error can't find the container with id 6f90f5e29907936e1e77544615459208b7796e502fd7ccae11f347b2111ef777 Nov 03 00:25:55.832980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.832959 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw"] Nov 03 00:25:55.926708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.926685 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:55.926708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.926712 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gk95\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-kube-api-access-6gk95\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:55.927136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:55.926725 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:25:56.368579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.368496 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" event={"ID":"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4","Type":"ContainerStarted","Data":"0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884"} Nov 03 00:25:56.368579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.368558 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" event={"ID":"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4","Type":"ContainerStarted","Data":"6f90f5e29907936e1e77544615459208b7796e502fd7ccae11f347b2111ef777"} Nov 03 00:25:56.368808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.368590 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:25:56.370189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.370162 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" event={"ID":"92e647e4-68b9-4659-967d-8f359de841b9","Type":"ContainerStarted","Data":"6d734fc178d8c5fd33ba9972004293fc219c1f1960e77b50ec65a42bc1b69f5c"} Nov 03 00:25:56.370321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.370211 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" event={"ID":"92e647e4-68b9-4659-967d-8f359de841b9","Type":"ContainerStarted","Data":"84eac5df5c3a8d665854a5851a684d6ea7f079404a9cf96a93c2a1eac33a54d2"} Nov 03 00:25:56.370321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.370264 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:25:56.371546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.371522 5144 generic.go:356] "Generic (PLEG): container finished" podID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerID="68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9" exitCode=0 Nov 03 00:25:56.371667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.371561 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" Nov 03 00:25:56.371667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.371569 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" event={"ID":"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3","Type":"ContainerDied","Data":"68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9"} Nov 03 00:25:56.371667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.371600 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-698cf49b84-t69zx" event={"ID":"ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3","Type":"ContainerDied","Data":"2552955b91ec2f06e1cacf34b110916c00853641b3fbccb81098993d0df2da7c"} Nov 03 00:25:56.371667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.371624 5144 scope.go:122] "RemoveContainer" containerID="68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9" Nov 03 00:25:56.383077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.382994 5144 scope.go:122] "RemoveContainer" containerID="68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9" Nov 03 00:25:56.383356 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:25:56.383330 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9\": container with ID starting with 68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9 not found: ID does not exist" containerID="68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9" Nov 03 00:25:56.383560 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.383359 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9"} err="failed to get container status \"68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9\": rpc error: code = NotFound desc = could not find container \"68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9\": container with ID starting with 68f9f2b2276c4972fb7c979f88d812166027f5857aaf542ea88e192e04083cd9 not found: ID does not exist" Nov 03 00:25:56.483006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.482956 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" podStartSLOduration=1.482940477 podStartE2EDuration="1.482940477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:56.481700514 +0000 UTC m=+1246.627394982" watchObservedRunningTime="2026-11-03 00:25:56.482940477 +0000 UTC m=+1246.628634934" Nov 03 00:25:56.596753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.596703 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" podStartSLOduration=2.5966882399999998 podStartE2EDuration="2.59668824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:56.594000554 +0000 UTC m=+1246.739695011" watchObservedRunningTime="2026-11-03 00:25:56.59668824 +0000 UTC m=+1246.742382696" Nov 03 00:25:56.709404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.709319 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-t69zx"] Nov 03 00:25:56.715403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:56.715372 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-698cf49b84-t69zx"] Nov 03 00:25:57.285674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:57.285645 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:25:57.286118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:57.286067 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:25:57.998099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:25:57.998070 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" path="/var/lib/kubelet/pods/ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3/volumes" Nov 03 00:26:07.384463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:07.384427 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:26:07.384938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:07.384510 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:26:10.602457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:10.602417 5144 scope.go:122] "RemoveContainer" containerID="f95211971d2eed3a45642e3219eab2b3bdae5c30f5d1e76dcc60d36cf4a0b63d" Nov 03 00:26:10.621259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:10.621228 5144 scope.go:122] "RemoveContainer" containerID="ff00670133ad83e38a25295ee8de7928f71d64835a05f50e99db17ba66ab47a1" Nov 03 00:26:10.635150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:10.635127 5144 scope.go:122] "RemoveContainer" containerID="42409b4884ecf86b480232c44028295e5e6c86abd101bcce0ce82cea0246c2c7" Nov 03 00:26:10.645680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:10.645659 5144 scope.go:122] "RemoveContainer" containerID="07f9ca3a072c42420337c8b79eb26b0f94cf1e4aba749c72b2870cbd641bd51f" Nov 03 00:26:10.657940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:10.657922 5144 scope.go:122] "RemoveContainer" containerID="8b5d816ef571533df0fa71541d37d287fd84ac22e2a36273849c9a2575d090f8" Nov 03 00:26:10.669213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:10.669187 5144 scope.go:122] "RemoveContainer" containerID="26a7af0a762426abce866d14600f0bf0a307386206383de678dda41c907ab55e" Nov 03 00:26:12.113774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.113742 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:26:12.114261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.113879 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:26:12.157157 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:26:12.157121 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:26:12.284694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.284656 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq"] Nov 03 00:26:12.285767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.285746 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerName="payload-processing" Nov 03 00:26:12.285819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.285773 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerName="payload-processing" Nov 03 00:26:12.286012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.285993 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba6dbe4d-9810-4fca-a8a5-f23ab8deaec3" containerName="payload-processing" Nov 03 00:26:12.309659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.294718 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.351613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.351491 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.351613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.351566 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.353106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.351992 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.353106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.352031 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-887tg\" (UniqueName: \"kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.353106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.352115 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.353106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.352229 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.362659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.362618 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq"] Nov 03 00:26:12.393615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.393524 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-531ee4\"/\"distinct-w2-531ee4-kserve-self-signed-certs\"" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.454046 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.454088 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.454173 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.454199 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-887tg\" (UniqueName: \"kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.454295 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.454449 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.455717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.455130 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.456518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.456141 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.457921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.457861 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.472263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.472229 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.472899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.472873 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.517666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.513169 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-887tg\" (UniqueName: \"kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg\") pod \"distinct-w2-531ee4-kserve-589bffb7b8-jrwmq\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:12.620572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:12.620546 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:13.006477 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:26:13.006452 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb9c2511d_43e0_4a2e_8730_d851ec97e70b.slice/crio-d19c8f6d595acefe2278211f18e8bf9aa645a7b07c1655e33d46a7c57f0febe6 WatchSource:0}: Error finding container d19c8f6d595acefe2278211f18e8bf9aa645a7b07c1655e33d46a7c57f0febe6: Status 404 returned error can't find the container with id d19c8f6d595acefe2278211f18e8bf9aa645a7b07c1655e33d46a7c57f0febe6 Nov 03 00:26:13.007238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.007192 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq"] Nov 03 00:26:13.640809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.640776 5144 generic.go:356] "Generic (PLEG): container finished" podID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerID="2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677" exitCode=0 Nov 03 00:26:13.641238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.640846 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" event={"ID":"dc275659-89d6-429c-8f7b-65976b66f0c9","Type":"ContainerDied","Data":"2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677"} Nov 03 00:26:13.641238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.641214 5144 scope.go:122] "RemoveContainer" containerID="2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677" Nov 03 00:26:13.642601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.642570 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:13.642718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.642664 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:13.642718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.642701 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:13.642933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.642909 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" event={"ID":"b9c2511d-43e0-4a2e-8730-d851ec97e70b","Type":"ContainerStarted","Data":"8a0e6c7f09703fb2b374c8cd379d012a4dc9e765c5d06fe094a8fa252584a987"} Nov 03 00:26:13.642984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.642952 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" event={"ID":"b9c2511d-43e0-4a2e-8730-d851ec97e70b","Type":"ContainerStarted","Data":"d19c8f6d595acefe2278211f18e8bf9aa645a7b07c1655e33d46a7c57f0febe6"} Nov 03 00:26:13.934375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:13.934316 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" podStartSLOduration=1.9342984159999999 podStartE2EDuration="1.934298416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:13.920247762 +0000 UTC m=+1264.065942280" watchObservedRunningTime="2026-11-03 00:26:13.934298416 +0000 UTC m=+1264.079992874" Nov 03 00:26:14.623672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:14.623224 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:14.653460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:14.653421 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" event={"ID":"dc275659-89d6-429c-8f7b-65976b66f0c9","Type":"ContainerStarted","Data":"af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f"} Nov 03 00:26:15.262313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:15.262274 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:26:15.262504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:15.262457 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:26:15.262551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:15.262509 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 00:26:16.263302 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:16.263268 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:26:16.263774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:16.263332 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 00:26:17.262682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:17.262645 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:26:17.262895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:17.262708 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 00:26:18.262935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:18.262892 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:26:18.263424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:18.262969 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 00:26:19.262594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:19.262555 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 00:26:19.262795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:19.262661 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 00:26:20.266259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:20.266227 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:26:20.266771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:20.266607 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:26:20.267401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:20.267378 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:26:22.621278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:22.621242 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:22.646427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:22.642226 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:22.737701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:22.737668 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 00:26:37.914047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:37.913973 5144 generic.go:356] "Generic (PLEG): container finished" podID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerID="845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51" exitCode=0 Nov 03 00:26:37.914047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:37.914015 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" event={"ID":"3795cc1f-3ad6-4676-a9b9-e36061737693","Type":"ContainerDied","Data":"845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51"} Nov 03 00:26:37.914473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:37.914325 5144 scope.go:122] "RemoveContainer" containerID="845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51" Nov 03 00:26:37.915641 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:37.915602 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:37.915702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:37.915688 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:37.915740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:37.915725 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:26:38.946568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:38.946535 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" event={"ID":"3795cc1f-3ad6-4676-a9b9-e36061737693","Type":"ContainerStarted","Data":"932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741"} Nov 03 00:26:39.360208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:39.360173 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:26:39.360415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:39.360339 5144 patch_prober.go:29] interesting pod/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:26:39.360415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:39.360385 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 00:26:40.360218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:40.360182 5144 patch_prober.go:29] interesting pod/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:26:40.360756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:40.360244 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 00:26:41.360771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:41.360733 5144 patch_prober.go:29] interesting pod/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:26:41.361662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:41.360795 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 00:26:42.164573 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:26:42.164540 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:26:42.360396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:42.360359 5144 patch_prober.go:29] interesting pod/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:26:42.360574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:42.360418 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 00:26:43.360251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:43.360209 5144 patch_prober.go:29] interesting pod/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 00:26:43.360686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:43.360267 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 00:26:44.364296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:44.364266 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:26:44.364766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:44.364746 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:26:44.365391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:26:44.365371 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:27:04.006065 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.006035 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq"] Nov 03 00:27:04.009826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.009808 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.070015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.069989 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-e671ea-openshift-default-dockercfg-jhg4h\"" Nov 03 00:27:04.169215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.169149 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.173964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.173716 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-468ts\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187001 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187074 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187114 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187167 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187235 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187265 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.189391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.187312 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.227595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.227512 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq"] Nov 03 00:27:04.289291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289188 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289260 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289287 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289319 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289350 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289366 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289387 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289425 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.289591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289458 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-468ts\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.290024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289815 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.290024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.289901 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.290134 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.290076 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.290189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.290162 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.290382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.290359 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.294062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.294042 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.294187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.294130 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.348044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.348011 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.348392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.348343 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-468ts\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts\") pod \"e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:04.626505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:04.626421 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:05.068908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.068883 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq"] Nov 03 00:27:05.071081 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:05.071054 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc360651c_0287_4473_9b64_8cbb2f8832d3.slice/crio-d7db607bd24f899b927afb3a8122d31918fcff8c73c7621d34933047a0bf986f WatchSource:0}: Error finding container d7db607bd24f899b927afb3a8122d31918fcff8c73c7621d34933047a0bf986f: Status 404 returned error can't find the container with id d7db607bd24f899b927afb3a8122d31918fcff8c73c7621d34933047a0bf986f Nov 03 00:27:05.073960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.073925 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:05.074043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.074008 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:05.074081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.074053 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:05.249709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.249319 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" event={"ID":"c360651c-0287-4473-9b64-8cbb2f8832d3","Type":"ContainerStarted","Data":"297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c"} Nov 03 00:27:05.249709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.249385 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" event={"ID":"c360651c-0287-4473-9b64-8cbb2f8832d3","Type":"ContainerStarted","Data":"d7db607bd24f899b927afb3a8122d31918fcff8c73c7621d34933047a0bf986f"} Nov 03 00:27:05.366126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.366035 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podStartSLOduration=2.366023133 podStartE2EDuration="2.366023133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:05.363485209 +0000 UTC m=+1315.509179664" watchObservedRunningTime="2026-11-03 00:27:05.366023133 +0000 UTC m=+1315.511717588" Nov 03 00:27:05.627579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:05.627495 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:06.633918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:06.633887 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:07.271951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:07.271916 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:07.272797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:07.272775 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:27:10.749332 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:10.749301 5144 scope.go:122] "RemoveContainer" containerID="ced1bb66d950cce96d6810baafbd740bbb5afb5725bd537964b1649d71b0f94d" Nov 03 00:27:11.872367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:11.872334 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq"] Nov 03 00:27:11.878315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:11.878293 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:11.939335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:11.939083 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq"] Nov 03 00:27:11.976701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:11.976576 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e671ea-dockercfg-twtwb\"" Nov 03 00:27:11.976701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:11.976664 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e671ea\"" Nov 03 00:27:12.034694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.019617 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jscpb\" (UniqueName: \"kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb\") pod \"payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.034694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.019753 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.124734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.121686 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jscpb\" (UniqueName: \"kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb\") pod \"payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.124734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.121789 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.124734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.122842 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.145693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.145127 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l"] Nov 03 00:27:12.151186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.150922 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.157535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.157505 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l"] Nov 03 00:27:12.172043 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:27:12.172015 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:27:12.180164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.180136 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jscpb\" (UniqueName: \"kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb\") pod \"payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.194074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.194053 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:12.228691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.223242 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.228691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.223409 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b28l5\" (UniqueName: \"kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5\") pod \"payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.324523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.324489 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.324720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.324571 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b28l5\" (UniqueName: \"kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5\") pod \"payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.325325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.325300 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.384361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.384299 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b28l5\" (UniqueName: \"kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5\") pod \"payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.470057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.470027 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:12.577120 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:12.577082 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7235f6d_4796_4b2c_af54_0f13aab7bbea.slice/crio-73b5e2d047c908de27722473cd7cd0650e2d1b9c8a54471f561b6aeb8e538225 WatchSource:0}: Error finding container 73b5e2d047c908de27722473cd7cd0650e2d1b9c8a54471f561b6aeb8e538225: Status 404 returned error can't find the container with id 73b5e2d047c908de27722473cd7cd0650e2d1b9c8a54471f561b6aeb8e538225 Nov 03 00:27:12.579536 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.579511 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq"] Nov 03 00:27:12.860074 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:12.860047 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde9530be_8f13_47e4_a5b9_d18c645d243f.slice/crio-6cfcd735182632f91099374e0d3ef7e96f422d4b24f51381d4958e51e8ecc7a8 WatchSource:0}: Error finding container 6cfcd735182632f91099374e0d3ef7e96f422d4b24f51381d4958e51e8ecc7a8: Status 404 returned error can't find the container with id 6cfcd735182632f91099374e0d3ef7e96f422d4b24f51381d4958e51e8ecc7a8 Nov 03 00:27:12.862430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:12.862401 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l"] Nov 03 00:27:13.329108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.329072 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" event={"ID":"de9530be-8f13-47e4-a5b9-d18c645d243f","Type":"ContainerStarted","Data":"386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356"} Nov 03 00:27:13.329658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.329131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" event={"ID":"de9530be-8f13-47e4-a5b9-d18c645d243f","Type":"ContainerStarted","Data":"6cfcd735182632f91099374e0d3ef7e96f422d4b24f51381d4958e51e8ecc7a8"} Nov 03 00:27:13.329658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.329152 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:13.331454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.331429 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" event={"ID":"a7235f6d-4796-4b2c-af54-0f13aab7bbea","Type":"ContainerStarted","Data":"8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c"} Nov 03 00:27:13.331578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.331478 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" event={"ID":"a7235f6d-4796-4b2c-af54-0f13aab7bbea","Type":"ContainerStarted","Data":"73b5e2d047c908de27722473cd7cd0650e2d1b9c8a54471f561b6aeb8e538225"} Nov 03 00:27:13.331578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.331531 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:13.464978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.464931 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" podStartSLOduration=2.464917954 podStartE2EDuration="2.464917954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:13.460391466 +0000 UTC m=+1323.606085922" watchObservedRunningTime="2026-11-03 00:27:13.464917954 +0000 UTC m=+1323.610612447" Nov 03 00:27:13.582572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:13.582478 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" podStartSLOduration=2.582463757 podStartE2EDuration="2.582463757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:13.580708997 +0000 UTC m=+1323.726403474" watchObservedRunningTime="2026-11-03 00:27:13.582463757 +0000 UTC m=+1323.728158257" Nov 03 00:27:14.433961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.433921 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg"] Nov 03 00:27:14.438852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.438825 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622235 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622302 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622367 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622406 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcmft\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622479 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622524 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622580 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622621 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.625984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.622756 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.675803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.647693 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-be578d-openshift-default-dockercfg-k7vqc\"" Nov 03 00:27:14.675803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.665485 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg"] Nov 03 00:27:14.724189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724096 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724142 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724173 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724194 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcmft\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724249 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724342 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724400 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724437 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724526 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724558 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724871 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.724953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.724937 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.725069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.725033 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.725258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.725091 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.727391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.727368 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.727624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.727605 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.782381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.781761 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcmft\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:14.782381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:14.782088 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token\") pod \"e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:15.058699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:15.058625 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:15.524667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:15.524624 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg"] Nov 03 00:27:15.563478 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:15.563412 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363102b0_1d5d_4235_8a0f_c7acca33b428.slice/crio-4a9d792c1a18e3f205b24bb37164d21604b749d15497ce4fcd9dfc91f353325a WatchSource:0}: Error finding container 4a9d792c1a18e3f205b24bb37164d21604b749d15497ce4fcd9dfc91f353325a: Status 404 returned error can't find the container with id 4a9d792c1a18e3f205b24bb37164d21604b749d15497ce4fcd9dfc91f353325a Nov 03 00:27:15.566094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:15.566061 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:15.566192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:15.566129 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:15.566192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:15.566160 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:16.392237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:16.392196 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" event={"ID":"363102b0-1d5d-4235-8a0f-c7acca33b428","Type":"ContainerStarted","Data":"bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47"} Nov 03 00:27:16.392431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:16.392265 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" event={"ID":"363102b0-1d5d-4235-8a0f-c7acca33b428","Type":"ContainerStarted","Data":"4a9d792c1a18e3f205b24bb37164d21604b749d15497ce4fcd9dfc91f353325a"} Nov 03 00:27:16.511772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:16.511721 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podStartSLOduration=2.511706053 podStartE2EDuration="2.511706053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:16.509575674 +0000 UTC m=+1326.655270153" watchObservedRunningTime="2026-11-03 00:27:16.511706053 +0000 UTC m=+1326.657400509" Nov 03 00:27:17.059663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:17.059606 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:17.064672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:17.064612 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:17.400775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:17.400702 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:17.401510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:17.401490 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:27:19.900969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:19.900941 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w"] Nov 03 00:27:19.905691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:19.905670 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:19.976031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:19.976006 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-fd6ea3-openshift-default-dockercfg-rs4x7\"" Nov 03 00:27:20.009910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.009872 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.009977 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010077 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010201 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010235 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010391 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010466 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.010556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.010540 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rzfp\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.030512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.030483 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w"] Nov 03 00:27:20.111704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.111677 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rzfp\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.111862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.111753 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.111951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.111857 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.111951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.111915 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.111965 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112005 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112029 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112113 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112164 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112173 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112232 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112250 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.112380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.112368 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.113097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.113070 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.114535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.114514 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.115002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.114973 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.182061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.181987 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.182219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.182107 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rzfp\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp\") pod \"e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.224557 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.224525 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:20.728456 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:20.728427 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a546bd_5baf_470c_9f9a_bddbf2ecead9.slice/crio-0610860be27a2dfc2df9ca3b4ad120aa1e56979eb237c129bf8b7d8893e0c2e3 WatchSource:0}: Error finding container 0610860be27a2dfc2df9ca3b4ad120aa1e56979eb237c129bf8b7d8893e0c2e3: Status 404 returned error can't find the container with id 0610860be27a2dfc2df9ca3b4ad120aa1e56979eb237c129bf8b7d8893e0c2e3 Nov 03 00:27:20.730050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.730028 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w"] Nov 03 00:27:20.731234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.731198 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:20.731324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.731287 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:20.731374 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:20.731328 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:27:21.438948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:21.438908 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" event={"ID":"d5a546bd-5baf-470c-9f9a-bddbf2ecead9","Type":"ContainerStarted","Data":"ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0"} Nov 03 00:27:21.439343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:21.438969 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" event={"ID":"d5a546bd-5baf-470c-9f9a-bddbf2ecead9","Type":"ContainerStarted","Data":"0610860be27a2dfc2df9ca3b4ad120aa1e56979eb237c129bf8b7d8893e0c2e3"} Nov 03 00:27:21.559286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:21.559237 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podStartSLOduration=2.559223375 podStartE2EDuration="2.559223375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:21.556373002 +0000 UTC m=+1331.702067487" watchObservedRunningTime="2026-11-03 00:27:21.559223375 +0000 UTC m=+1331.704917830" Nov 03 00:27:22.225086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:22.225006 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:22.230975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:22.230948 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:22.447703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:22.447672 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:22.448725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:22.448702 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:27:23.073612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.073574 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp"] Nov 03 00:27:23.078350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.078322 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.130136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.130114 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-be578d\"" Nov 03 00:27:23.130723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.130114 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-be578d-dockercfg-cmqz9\"" Nov 03 00:27:23.133480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.133458 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp"] Nov 03 00:27:23.433218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.433140 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.433363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.433340 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2cvt\" (UniqueName: \"kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt\") pod \"payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.534461 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.534427 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2cvt\" (UniqueName: \"kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt\") pod \"payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.534914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.534554 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.535264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.535245 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.593529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.593499 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2cvt\" (UniqueName: \"kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt\") pod \"payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:23.693116 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:23.693052 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:24.079907 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:24.079875 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod694a220a_db40_4d48_a822_b2d70894efd3.slice/crio-1c6a79ad8385bd96c833fe4ee75f7238954b55088f7e33966cf5b74239453a49 WatchSource:0}: Error finding container 1c6a79ad8385bd96c833fe4ee75f7238954b55088f7e33966cf5b74239453a49: Status 404 returned error can't find the container with id 1c6a79ad8385bd96c833fe4ee75f7238954b55088f7e33966cf5b74239453a49 Nov 03 00:27:24.080762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.080744 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp"] Nov 03 00:27:24.345672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.345573 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:27:24.345672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.345660 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:27:24.464737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.464706 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" event={"ID":"694a220a-db40-4d48-a822-b2d70894efd3","Type":"ContainerStarted","Data":"1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009"} Nov 03 00:27:24.464918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.464765 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" event={"ID":"694a220a-db40-4d48-a822-b2d70894efd3","Type":"ContainerStarted","Data":"1c6a79ad8385bd96c833fe4ee75f7238954b55088f7e33966cf5b74239453a49"} Nov 03 00:27:24.464918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.464794 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:24.695902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:24.695706 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" podStartSLOduration=1.6956875409999999 podStartE2EDuration="1.695687541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:24.69109501 +0000 UTC m=+1334.836789466" watchObservedRunningTime="2026-11-03 00:27:24.695687541 +0000 UTC m=+1334.841381998" Nov 03 00:27:28.294654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.292722 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm"] Nov 03 00:27:28.325959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.325928 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.363029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.361584 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm"] Nov 03 00:27:28.378825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.378797 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-fd6ea3-dockercfg-f495d\"" Nov 03 00:27:28.380035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.380011 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-fd6ea3\"" Nov 03 00:27:28.431991 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.431963 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.432154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.432043 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjx6w\" (UniqueName: \"kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w\") pod \"payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.533154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.533120 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.533333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.533193 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjx6w\" (UniqueName: \"kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w\") pod \"payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.534054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.534030 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.591224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.591162 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjx6w\" (UniqueName: \"kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w\") pod \"payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:28.660322 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:28.660291 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:29.044796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:29.044770 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm"] Nov 03 00:27:29.045440 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:27:29.045412 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d420a25_f922_4d04_94bf_01ff94d4e71c.slice/crio-e54818c370f1533f5b960dd1fcc100f9aa9b945d78b01664d6b535c2d5351c7f WatchSource:0}: Error finding container e54818c370f1533f5b960dd1fcc100f9aa9b945d78b01664d6b535c2d5351c7f: Status 404 returned error can't find the container with id e54818c370f1533f5b960dd1fcc100f9aa9b945d78b01664d6b535c2d5351c7f Nov 03 00:27:29.509100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:29.509054 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" event={"ID":"2d420a25-f922-4d04-94bf-01ff94d4e71c","Type":"ContainerStarted","Data":"432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06"} Nov 03 00:27:29.509478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:29.509128 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" event={"ID":"2d420a25-f922-4d04-94bf-01ff94d4e71c","Type":"ContainerStarted","Data":"e54818c370f1533f5b960dd1fcc100f9aa9b945d78b01664d6b535c2d5351c7f"} Nov 03 00:27:29.509478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:29.509158 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:29.630769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:29.630703 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" podStartSLOduration=1.630684773 podStartE2EDuration="1.630684773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:29.621943704 +0000 UTC m=+1339.767638162" watchObservedRunningTime="2026-11-03 00:27:29.630684773 +0000 UTC m=+1339.776379230" Nov 03 00:27:35.476082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:35.476053 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:27:40.520712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:27:40.520673 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:27:42.157310 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:27:42.157268 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:28:05.048597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:05.048559 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm"] Nov 03 00:28:05.098830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:05.098797 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7fb3217d-8dncm"] Nov 03 00:28:05.998225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:05.998192 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb" path="/var/lib/kubelet/pods/42b0a0d9-ddad-4dc3-a4a6-8698656f2bcb/volumes" Nov 03 00:28:12.158265 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:28:12.158225 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:28:22.131855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:22.115140 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7"] Nov 03 00:28:22.183226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:22.183184 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec751bd2-q6js7"] Nov 03 00:28:24.001141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:24.001097 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdf2d58-007e-485f-9634-00e72b118cdd" path="/var/lib/kubelet/pods/2cdf2d58-007e-485f-9634-00e72b118cdd/volumes" Nov 03 00:28:28.961203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:28.961164 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl"] Nov 03 00:28:28.966280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:28.966257 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.019822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.019789 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl"] Nov 03 00:28:29.064126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.062194 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:28:29.064126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.062986 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:28:29.103041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.103003 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx9hw\" (UniqueName: \"kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.103277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.103251 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.103371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.103354 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.204865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.204834 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.205038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.204904 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.205089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.205059 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx9hw\" (UniqueName: \"kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.205699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.205626 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.207597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.207567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.262816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.262783 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx9hw\" (UniqueName: \"kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw\") pod \"maas-api-revoke-sub-f5ac2d31-cj7gl\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.281721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.281699 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:29.700987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:29.684314 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl"] Nov 03 00:28:29.700987 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:28:29.690100 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15b3b8cd_8aea_4c21_a1e9_5ba4c9b13bfe.slice/crio-9a0a0f584079faae8eb3d5f657403edabc674ed349861e557e75cdb3260d2caa WatchSource:0}: Error finding container 9a0a0f584079faae8eb3d5f657403edabc674ed349861e557e75cdb3260d2caa: Status 404 returned error can't find the container with id 9a0a0f584079faae8eb3d5f657403edabc674ed349861e557e75cdb3260d2caa Nov 03 00:28:30.215156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:30.215111 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" event={"ID":"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe","Type":"ContainerStarted","Data":"9a0a0f584079faae8eb3d5f657403edabc674ed349861e557e75cdb3260d2caa"} Nov 03 00:28:31.229261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:31.229184 5144 generic.go:356] "Generic (PLEG): container finished" podID="15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" containerID="a2f327bcadd7a2590f077f0f84731825275a85be19d08bf5ffcf7bd29d805216" exitCode=0 Nov 03 00:28:31.229697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:31.229258 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" event={"ID":"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe","Type":"ContainerDied","Data":"a2f327bcadd7a2590f077f0f84731825275a85be19d08bf5ffcf7bd29d805216"} Nov 03 00:28:32.544878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.544848 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:32.596554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.596513 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca\") pod \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " Nov 03 00:28:32.596806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.596688 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw\" (UniqueName: \"kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw\") pod \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " Nov 03 00:28:32.596943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.596921 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca\") pod \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\" (UID: \"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe\") " Nov 03 00:28:32.597363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.597337 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca" pod "15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" (UID: "15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:28:32.597958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.597933 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:32.617968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.617929 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca" pod "15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" (UID: "15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:28:32.618136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.617993 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw" pod "15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" (UID: "15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe"). InnerVolumeSpecName "kube-api-access-mx9hw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:28:32.699044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.699009 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:32.699044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:32.699046 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx9hw\" (UniqueName: \"kubernetes.io/projected/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe-kube-api-access-mx9hw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:28:33.267487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:33.267438 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" event={"ID":"15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe","Type":"ContainerDied","Data":"9a0a0f584079faae8eb3d5f657403edabc674ed349861e557e75cdb3260d2caa"} Nov 03 00:28:33.268878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:33.267546 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a0a0f584079faae8eb3d5f657403edabc674ed349861e557e75cdb3260d2caa" Nov 03 00:28:33.268878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:33.267704 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl" Nov 03 00:28:33.330237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:33.330044 5144 patch_prober.go:29] interesting pod/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": read tcp 10.132.0.2:48378->10.132.0.143:15021: read: connection reset by peer" start-of-body= Nov 03 00:28:33.330237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:33.330132 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": read tcp 10.132.0.2:48378->10.132.0.143:15021: read: connection reset by peer" Nov 03 00:28:34.277151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.277119 5144 generic.go:356] "Generic (PLEG): container finished" podID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerID="bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47" exitCode=0 Nov 03 00:28:34.277585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.277188 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" event={"ID":"363102b0-1d5d-4235-8a0f-c7acca33b428","Type":"ContainerDied","Data":"bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47"} Nov 03 00:28:34.277747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.277731 5144 scope.go:122] "RemoveContainer" containerID="bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47" Nov 03 00:28:34.279124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.279104 5144 generic.go:356] "Generic (PLEG): container finished" podID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerID="297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c" exitCode=0 Nov 03 00:28:34.279225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.279182 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" event={"ID":"c360651c-0287-4473-9b64-8cbb2f8832d3","Type":"ContainerDied","Data":"297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c"} Nov 03 00:28:34.279288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.279259 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:34.279353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.279327 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:34.279414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.279375 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:34.279586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.279569 5144 scope.go:122] "RemoveContainer" containerID="297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c" Nov 03 00:28:34.280900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.280874 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:34.281000 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.280945 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:34.281061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:34.280995 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:35.289979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:35.289941 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" event={"ID":"c360651c-0287-4473-9b64-8cbb2f8832d3","Type":"ContainerStarted","Data":"26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51"} Nov 03 00:28:35.292404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:35.292379 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" event={"ID":"363102b0-1d5d-4235-8a0f-c7acca33b428","Type":"ContainerStarted","Data":"192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed"} Nov 03 00:28:35.651719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:35.634215 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:28:35.651719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:35.646767 5144 patch_prober.go:29] interesting pod/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 00:28:35.651719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:35.646885 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 00:28:36.059658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.059017 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:28:36.059658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.059130 5144 patch_prober.go:29] interesting pod/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:28:36.059658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.059170 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 00:28:36.068971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.068947 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l"] Nov 03 00:28:36.074136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.074108 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1112f7b8-wcb4l"] Nov 03 00:28:36.639654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.637155 5144 patch_prober.go:29] interesting pod/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 00:28:36.639654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:36.637219 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 00:28:37.064659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:37.060048 5144 patch_prober.go:29] interesting pod/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:28:37.064659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:37.060113 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 00:28:37.272103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:37.272049 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:28:37.636659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:37.634164 5144 patch_prober.go:29] interesting pod/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 00:28:37.636659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:37.634226 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 00:28:38.001138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.001058 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37aa31bc-faf8-4268-97ba-16114895693e" path="/var/lib/kubelet/pods/37aa31bc-faf8-4268-97ba-16114895693e/volumes" Nov 03 00:28:38.063657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.060028 5144 patch_prober.go:29] interesting pod/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:28:38.063657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.060092 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 00:28:38.110103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.110070 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4"] Nov 03 00:28:38.178754 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.178232 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7a19e6f-tnhj4"] Nov 03 00:28:38.630657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.630038 5144 patch_prober.go:29] interesting pod/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 00:28:38.630657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:38.630104 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 00:28:39.075658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:39.071763 5144 patch_prober.go:29] interesting pod/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:28:39.075658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:39.071844 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 00:28:39.629211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:39.629154 5144 patch_prober.go:29] interesting pod/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body= Nov 03 00:28:39.629547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:39.629516 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" Nov 03 00:28:39.998751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:39.998672 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ee6458a-af63-45f7-b542-78df123591ec" path="/var/lib/kubelet/pods/1ee6458a-af63-45f7-b542-78df123591ec/volumes" Nov 03 00:28:40.061077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:40.061036 5144 patch_prober.go:29] interesting pod/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body= Nov 03 00:28:40.062154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:40.062121 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" Nov 03 00:28:40.631956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:40.631927 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:28:40.633263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:40.633240 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:28:41.063211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:41.063178 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:28:41.063682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:41.063660 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:28:41.064450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:41.064433 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:28:42.159730 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:28:42.159697 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:28:50.047919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:50.047879 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68"] Nov 03 00:28:50.095965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:50.095935 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd859c9c-bdk68"] Nov 03 00:28:52.029183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:52.029144 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0a72fc-f718-4e55-ab56-0f62f4bae131" path="/var/lib/kubelet/pods/1e0a72fc-f718-4e55-ab56-0f62f4bae131/volumes" Nov 03 00:28:53.049077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:53.049043 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql"] Nov 03 00:28:53.099373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:53.099341 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bdec8d5-khbql"] Nov 03 00:28:53.997864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:53.997830 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a5e3786-63f4-4a67-910c-53320c5e1879" path="/var/lib/kubelet/pods/2a5e3786-63f4-4a67-910c-53320c5e1879/volumes" Nov 03 00:28:58.234357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.234246 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k"] Nov 03 00:28:58.235328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.235306 5144 cpu_manager.go:414] "containerMap: removing container" podUID="15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" containerName="revoke-keys" Nov 03 00:28:58.235416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.235330 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" containerName="revoke-keys" Nov 03 00:28:58.235589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.235575 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" containerName="revoke-keys" Nov 03 00:28:58.238771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.238748 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271490 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271525 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271547 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85vgh\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271577 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271752 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271808 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.271986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.271896 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.272064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.272044 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.272125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.272070 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.291042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.291010 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-531ee4-openshift-default-dockercfg-pcjd8\"" Nov 03 00:28:58.366694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.364232 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k"] Nov 03 00:28:58.373160 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373135 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373251 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373270 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373330 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373457 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373503 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85vgh\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373541 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373620 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373683 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.373812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.373751 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.374100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.374078 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.374269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.374157 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.374269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.374179 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.374269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.374245 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.376572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.376519 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.377025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.377005 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.432280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.432248 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.432598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.432574 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85vgh\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.553572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.553544 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:28:58.992992 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:28:58.992965 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3499f53_e055_4cc9_96a5_8a28e18dc3b9.slice/crio-a6b6f708af83378426e720827b01948eb432bad410f0e9a3a6f8f5277a820948 WatchSource:0}: Error finding container a6b6f708af83378426e720827b01948eb432bad410f0e9a3a6f8f5277a820948: Status 404 returned error can't find the container with id a6b6f708af83378426e720827b01948eb432bad410f0e9a3a6f8f5277a820948 Nov 03 00:28:58.993131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.993101 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k"] Nov 03 00:28:58.995569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.995538 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:58.995692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.995595 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:58.995692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:58.995649 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:28:59.674883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:59.674831 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" event={"ID":"b3499f53-e055-4cc9-96a5-8a28e18dc3b9","Type":"ContainerStarted","Data":"eb38b2911e2feb3bafbcf813c7a628f529c2a2606b5898e339435e00ee2b648c"} Nov 03 00:28:59.675327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:59.674898 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" event={"ID":"b3499f53-e055-4cc9-96a5-8a28e18dc3b9","Type":"ContainerStarted","Data":"a6b6f708af83378426e720827b01948eb432bad410f0e9a3a6f8f5277a820948"} Nov 03 00:28:59.846094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:28:59.846043 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podStartSLOduration=1.8460263590000001 podStartE2EDuration="1.846026359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:59.844825339 +0000 UTC m=+1429.990519793" watchObservedRunningTime="2026-11-03 00:28:59.846026359 +0000 UTC m=+1429.991720815" Nov 03 00:29:00.564661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:00.558073 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:29:00.564661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:00.561816 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:29:00.564661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:00.561873 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:29:01.554725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:01.554682 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:29:01.555151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:01.554778 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:29:02.554905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.554871 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:29:02.556249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.554926 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:29:02.709158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.709122 5144 generic.go:356] "Generic (PLEG): container finished" podID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerID="ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0" exitCode=0 Nov 03 00:29:02.709158 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.709152 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" event={"ID":"d5a546bd-5baf-470c-9f9a-bddbf2ecead9","Type":"ContainerDied","Data":"ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0"} Nov 03 00:29:02.709708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.709677 5144 scope.go:122] "RemoveContainer" containerID="ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0" Nov 03 00:29:02.711440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.711413 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:02.711516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.711481 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:02.711561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:02.711516 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:03.566156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:03.566096 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:29:03.566732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:03.566187 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:29:03.757654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:03.756257 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" event={"ID":"d5a546bd-5baf-470c-9f9a-bddbf2ecead9","Type":"ContainerStarted","Data":"5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef"} Nov 03 00:29:04.071647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.071386 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl"] Nov 03 00:29:04.076097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.076066 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db523f03-tr6zl"] Nov 03 00:29:04.238655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.237995 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:29:04.238655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.238127 5144 patch_prober.go:29] interesting pod/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 00:29:04.238655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.238168 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 00:29:04.563397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.563361 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:29:04.563718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:04.563690 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:29:05.259661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:05.259362 5144 patch_prober.go:29] interesting pod/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 00:29:05.259661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:05.259435 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 00:29:05.554422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:05.554346 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:29:05.554422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:05.554409 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:29:06.004390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.004362 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598ed79f-b24b-4ba9-ab94-a4064b0630d8" path="/var/lib/kubelet/pods/598ed79f-b24b-4ba9-ab94-a4064b0630d8/volumes" Nov 03 00:29:06.056029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.055996 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc"] Nov 03 00:29:06.100030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.100001 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-293a3dfd-m5tqc"] Nov 03 00:29:06.231656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.226358 5144 patch_prober.go:29] interesting pod/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 00:29:06.231656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.226522 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 00:29:06.891934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.890906 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:29:06.891934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.891809 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:29:06.892340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:06.891954 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:29:07.247083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:07.246998 5144 patch_prober.go:29] interesting pod/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 00:29:07.247083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:07.247056 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 00:29:07.447963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:07.447936 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:29:07.997792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:07.997757 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de2c310-0020-48f5-a718-a2ab673e0f65" path="/var/lib/kubelet/pods/7de2c310-0020-48f5-a718-a2ab673e0f65/volumes" Nov 03 00:29:08.238100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:08.238053 5144 patch_prober.go:29] interesting pod/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 00:29:08.238406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:08.238376 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 00:29:08.466310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:08.466282 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks"] Nov 03 00:29:08.579998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:08.579967 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft"] Nov 03 00:29:09.230499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:09.230466 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:29:09.231624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:09.231603 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:29:10.791984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:10.791958 5144 scope.go:122] "RemoveContainer" containerID="5c17cae6217cf824870a8c034a32ab038613652012610f8f2fa3fb310b0b7ca3" Nov 03 00:29:11.156782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.156701 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm"] Nov 03 00:29:11.161298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.161276 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.247261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.247226 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm"] Nov 03 00:29:11.252116 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.252077 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.252555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.252531 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.252727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.252710 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4vk5\" (UniqueName: \"kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.304655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.302917 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:29:11.304655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.303063 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:29:11.358366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.358337 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.358670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.358614 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4vk5\" (UniqueName: \"kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.359028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.358983 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.360578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.360558 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.361778 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.361757 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.415586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.415514 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4vk5\" (UniqueName: \"kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.477876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.477846 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:11.854192 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:29:11.854164 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae77c19e_dd79_47cb_8853_5978a9650419.slice/crio-8f6d9182e51b61a839b2f5004fde255a55c9163c03b49d8e9eb47cd8adde1fdf WatchSource:0}: Error finding container 8f6d9182e51b61a839b2f5004fde255a55c9163c03b49d8e9eb47cd8adde1fdf: Status 404 returned error can't find the container with id 8f6d9182e51b61a839b2f5004fde255a55c9163c03b49d8e9eb47cd8adde1fdf Nov 03 00:29:11.856003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:11.855978 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm"] Nov 03 00:29:12.182373 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:12.182273 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:29:12.871311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:12.871274 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" event={"ID":"ae77c19e-dd79-47cb-8853-5978a9650419","Type":"ContainerStarted","Data":"bb7668f7ee4ec5c986384fce2df50c766ba6f68a65ca18f1858269e8cb8f40b2"} Nov 03 00:29:12.871869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:12.871338 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" event={"ID":"ae77c19e-dd79-47cb-8853-5978a9650419","Type":"ContainerStarted","Data":"8f6d9182e51b61a839b2f5004fde255a55c9163c03b49d8e9eb47cd8adde1fdf"} Nov 03 00:29:12.988486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:12.988436 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" podStartSLOduration=1.988423555 podStartE2EDuration="1.988423555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:12.985681321 +0000 UTC m=+1443.131375779" watchObservedRunningTime="2026-11-03 00:29:12.988423555 +0000 UTC m=+1443.134118010" Nov 03 00:29:13.467568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.467521 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" podUID="64c9bbfd-1370-4771-8db1-3cac8c9a93ce" containerName="payload-processing" containerID="cri-o://3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c" gracePeriod=25 Nov 03 00:29:13.581888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.580565 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" podUID="c5bef802-1658-472f-9df7-3a8d88bf6c07" containerName="payload-pre-processing" containerID="cri-o://2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57" gracePeriod=25 Nov 03 00:29:13.802975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.802951 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:29:13.882061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.882027 5144 generic.go:356] "Generic (PLEG): container finished" podID="64c9bbfd-1370-4771-8db1-3cac8c9a93ce" containerID="3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c" exitCode=0 Nov 03 00:29:13.882470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.882073 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" Nov 03 00:29:13.882470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.882112 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" event={"ID":"64c9bbfd-1370-4771-8db1-3cac8c9a93ce","Type":"ContainerDied","Data":"3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c"} Nov 03 00:29:13.882470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.882169 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks" event={"ID":"64c9bbfd-1370-4771-8db1-3cac8c9a93ce","Type":"ContainerDied","Data":"e2535d2d966cbbb9532279d96a752617685ada7d7e734cf87238d9d1b8ca6b3a"} Nov 03 00:29:13.882470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.882193 5144 scope.go:122] "RemoveContainer" containerID="3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c" Nov 03 00:29:13.883459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.883443 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:29:13.885187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.885160 5144 generic.go:356] "Generic (PLEG): container finished" podID="ae77c19e-dd79-47cb-8853-5978a9650419" containerID="bb7668f7ee4ec5c986384fce2df50c766ba6f68a65ca18f1858269e8cb8f40b2" exitCode=0 Nov 03 00:29:13.885364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.885349 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" event={"ID":"ae77c19e-dd79-47cb-8853-5978a9650419","Type":"ContainerDied","Data":"bb7668f7ee4ec5c986384fce2df50c766ba6f68a65ca18f1858269e8cb8f40b2"} Nov 03 00:29:13.887122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.887104 5144 generic.go:356] "Generic (PLEG): container finished" podID="c5bef802-1658-472f-9df7-3a8d88bf6c07" containerID="2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57" exitCode=0 Nov 03 00:29:13.887236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.887146 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" Nov 03 00:29:13.887236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.887191 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" event={"ID":"c5bef802-1658-472f-9df7-3a8d88bf6c07","Type":"ContainerDied","Data":"2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57"} Nov 03 00:29:13.887236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.887230 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft" event={"ID":"c5bef802-1658-472f-9df7-3a8d88bf6c07","Type":"ContainerDied","Data":"eccfcbf5bb76dda76554ce2373c19c3342dfd062beef7a657fdc3a1c8a5b2af5"} Nov 03 00:29:13.895036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.895019 5144 scope.go:122] "RemoveContainer" containerID="3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c" Nov 03 00:29:13.895279 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:13.895261 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c\": container with ID starting with 3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c not found: ID does not exist" containerID="3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c" Nov 03 00:29:13.895322 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.895287 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c"} err="failed to get container status \"3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c\": rpc error: code = NotFound desc = could not find container \"3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c\": container with ID starting with 3dc16ddbf6588cbb970f8f639d64329f6e794bf760ea35fdb8be086e33b7038c not found: ID does not exist" Nov 03 00:29:13.895322 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.895303 5144 scope.go:122] "RemoveContainer" containerID="2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57" Nov 03 00:29:13.897717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.897699 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume\") pod \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " Nov 03 00:29:13.897785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.897760 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6\" (UniqueName: \"kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6\") pod \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\" (UID: \"64c9bbfd-1370-4771-8db1-3cac8c9a93ce\") " Nov 03 00:29:13.898052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.898034 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume" pod "64c9bbfd-1370-4771-8db1-3cac8c9a93ce" (UID: "64c9bbfd-1370-4771-8db1-3cac8c9a93ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:13.898243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.898229 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:13.900063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.900043 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6" pod "64c9bbfd-1370-4771-8db1-3cac8c9a93ce" (UID: "64c9bbfd-1370-4771-8db1-3cac8c9a93ce"). InnerVolumeSpecName "kube-api-access-cmvs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:13.906090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.906063 5144 scope.go:122] "RemoveContainer" containerID="2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57" Nov 03 00:29:13.906362 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:13.906340 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57\": container with ID starting with 2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57 not found: ID does not exist" containerID="2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57" Nov 03 00:29:13.906454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.906367 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57"} err="failed to get container status \"2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57\": rpc error: code = NotFound desc = could not find container \"2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57\": container with ID starting with 2296b5c8ac3ef1146409dd775eac3e9656541a688a7ef4e3171158786c054b57 not found: ID does not exist" Nov 03 00:29:13.998764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.998731 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5\" (UniqueName: \"kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5\") pod \"c5bef802-1658-472f-9df7-3a8d88bf6c07\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " Nov 03 00:29:13.998970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.998953 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume\") pod \"c5bef802-1658-472f-9df7-3a8d88bf6c07\" (UID: \"c5bef802-1658-472f-9df7-3a8d88bf6c07\") " Nov 03 00:29:13.999386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.999355 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume" pod "c5bef802-1658-472f-9df7-3a8d88bf6c07" (UID: "c5bef802-1658-472f-9df7-3a8d88bf6c07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:13.999500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:13.999428 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmvs6\" (UniqueName: \"kubernetes.io/projected/64c9bbfd-1370-4771-8db1-3cac8c9a93ce-kube-api-access-cmvs6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:14.001030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.001004 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5" pod "c5bef802-1658-472f-9df7-3a8d88bf6c07" (UID: "c5bef802-1658-472f-9df7-3a8d88bf6c07"). InnerVolumeSpecName "kube-api-access-hgfv5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:14.100700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.100586 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5bef802-1658-472f-9df7-3a8d88bf6c07-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:14.100700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.100616 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgfv5\" (UniqueName: \"kubernetes.io/projected/c5bef802-1658-472f-9df7-3a8d88bf6c07-kube-api-access-hgfv5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:14.305920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.305892 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks"] Nov 03 00:29:14.310404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.310384 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a9404c-79684694cc-wbnks"] Nov 03 00:29:14.418748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.418673 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft"] Nov 03 00:29:14.424391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:14.424366 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a9404c-7ff76bb595-nbdft"] Nov 03 00:29:15.066100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.066074 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:15.116476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.116448 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca\") pod \"ae77c19e-dd79-47cb-8853-5978a9650419\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " Nov 03 00:29:15.116603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.116493 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca\") pod \"ae77c19e-dd79-47cb-8853-5978a9650419\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " Nov 03 00:29:15.116695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.116654 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5\" (UniqueName: \"kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5\") pod \"ae77c19e-dd79-47cb-8853-5978a9650419\" (UID: \"ae77c19e-dd79-47cb-8853-5978a9650419\") " Nov 03 00:29:15.116868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.116843 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca" pod "ae77c19e-dd79-47cb-8853-5978a9650419" (UID: "ae77c19e-dd79-47cb-8853-5978a9650419"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:15.117848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.117822 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:15.119408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.119377 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca" pod "ae77c19e-dd79-47cb-8853-5978a9650419" (UID: "ae77c19e-dd79-47cb-8853-5978a9650419"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:15.119543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.119384 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5" pod "ae77c19e-dd79-47cb-8853-5978a9650419" (UID: "ae77c19e-dd79-47cb-8853-5978a9650419"). InnerVolumeSpecName "kube-api-access-j4vk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:15.219797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.219725 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae77c19e-dd79-47cb-8853-5978a9650419-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:15.219797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.219758 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4vk5\" (UniqueName: \"kubernetes.io/projected/ae77c19e-dd79-47cb-8853-5978a9650419-kube-api-access-j4vk5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:15.917740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.917708 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" Nov 03 00:29:15.917928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.917704 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm" event={"ID":"ae77c19e-dd79-47cb-8853-5978a9650419","Type":"ContainerDied","Data":"8f6d9182e51b61a839b2f5004fde255a55c9163c03b49d8e9eb47cd8adde1fdf"} Nov 03 00:29:15.917928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.917866 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f6d9182e51b61a839b2f5004fde255a55c9163c03b49d8e9eb47cd8adde1fdf" Nov 03 00:29:15.999484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.999456 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c9bbfd-1370-4771-8db1-3cac8c9a93ce" path="/var/lib/kubelet/pods/64c9bbfd-1370-4771-8db1-3cac8c9a93ce/volumes" Nov 03 00:29:15.999864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:15.999849 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5bef802-1658-472f-9df7-3a8d88bf6c07" path="/var/lib/kubelet/pods/c5bef802-1658-472f-9df7-3a8d88bf6c07/volumes" Nov 03 00:29:16.866402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.866367 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72"] Nov 03 00:29:16.867130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867114 5144 cpu_manager.go:414] "containerMap: removing container" podUID="64c9bbfd-1370-4771-8db1-3cac8c9a93ce" containerName="payload-processing" Nov 03 00:29:16.867130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867131 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c9bbfd-1370-4771-8db1-3cac8c9a93ce" containerName="payload-processing" Nov 03 00:29:16.867281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867152 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c5bef802-1658-472f-9df7-3a8d88bf6c07" containerName="payload-pre-processing" Nov 03 00:29:16.867281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867158 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5bef802-1658-472f-9df7-3a8d88bf6c07" containerName="payload-pre-processing" Nov 03 00:29:16.867281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867169 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ae77c19e-dd79-47cb-8853-5978a9650419" containerName="revoke-keys" Nov 03 00:29:16.867281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867175 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae77c19e-dd79-47cb-8853-5978a9650419" containerName="revoke-keys" Nov 03 00:29:16.867472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867304 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5bef802-1658-472f-9df7-3a8d88bf6c07" containerName="payload-pre-processing" Nov 03 00:29:16.867472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867312 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae77c19e-dd79-47cb-8853-5978a9650419" containerName="revoke-keys" Nov 03 00:29:16.867472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.867327 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c9bbfd-1370-4771-8db1-3cac8c9a93ce" containerName="payload-processing" Nov 03 00:29:16.870742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.870726 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.945533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.945496 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.945877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.945856 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946008 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946174 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946270 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946408 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pg47\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946536 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946748 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.946867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.946853 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:16.972803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:16.972770 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72"] Nov 03 00:29:17.017016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.016982 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-38f5a3-openshift-default-dockercfg-hnmsh\"" Nov 03 00:29:17.059603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.059559 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pg47\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.059890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.059872 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.060131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.060116 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.060246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.060232 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.060397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.060384 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.082655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.060577 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.082988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.082960 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.083244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.083223 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.083410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.083389 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.084481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.084458 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.110656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.084920 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.110916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.089040 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.111046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.089265 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.111147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.089902 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.124706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.113005 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.149303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.149275 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pg47\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.180658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.180603 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.184658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.184565 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy\") pod \"e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.483613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.483524 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:17.917739 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:29:17.917709 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6f913c4_cfcf_4f8a_96ef_66f0debc631f.slice/crio-6fc76c3f631e6d48d94505adf7303beb3ca028848edc2eef1e4cdc186afeae14 WatchSource:0}: Error finding container 6fc76c3f631e6d48d94505adf7303beb3ca028848edc2eef1e4cdc186afeae14: Status 404 returned error can't find the container with id 6fc76c3f631e6d48d94505adf7303beb3ca028848edc2eef1e4cdc186afeae14 Nov 03 00:29:17.918885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.918864 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72"] Nov 03 00:29:17.920882 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.920834 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:17.920975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.920926 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:17.921014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.920971 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:29:17.952862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:17.952828 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" event={"ID":"e6f913c4-cfcf-4f8a-96ef-66f0debc631f","Type":"ContainerStarted","Data":"6fc76c3f631e6d48d94505adf7303beb3ca028848edc2eef1e4cdc186afeae14"} Nov 03 00:29:18.963081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:18.963043 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" event={"ID":"e6f913c4-cfcf-4f8a-96ef-66f0debc631f","Type":"ContainerStarted","Data":"ae33c85f3e629291fdedc712152ee66db826d970f60cfab53a48746b159dc915"} Nov 03 00:29:19.113949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:19.113891 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podStartSLOduration=3.113871368 podStartE2EDuration="3.113871368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:19.107420155 +0000 UTC m=+1449.253114629" watchObservedRunningTime="2026-11-03 00:29:19.113871368 +0000 UTC m=+1449.259565824" Nov 03 00:29:19.491790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:19.491759 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:19.506650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:19.503938 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:19.973901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:19.973870 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:19.975067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:19.975045 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:29:20.049281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:20.049253 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974"] Nov 03 00:29:20.081348 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:20.081308 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b96e18f-l9974"] Nov 03 00:29:21.997103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:21.997072 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f48a828-89b9-4c39-b6ea-7c9d5577bd0a" path="/var/lib/kubelet/pods/3f48a828-89b9-4c39-b6ea-7c9d5577bd0a/volumes" Nov 03 00:29:24.048311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:24.048279 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8"] Nov 03 00:29:24.098015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:24.097987 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc92b52a-cjvz8"] Nov 03 00:29:25.998464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:25.998430 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44abe098-1c18-4929-89e6-24ead8738985" path="/var/lib/kubelet/pods/44abe098-1c18-4929-89e6-24ead8738985/volumes" Nov 03 00:29:30.194277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.194240 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps"] Nov 03 00:29:30.197902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.197879 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.253647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.253603 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps"] Nov 03 00:29:30.336653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.334660 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-38f5a3\"" Nov 03 00:29:30.336653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.334850 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-38f5a3-dockercfg-fhdw8\"" Nov 03 00:29:30.473713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.470040 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.473713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.470156 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tlrt\" (UniqueName: \"kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.571002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.570972 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.571170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.571070 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tlrt\" (UniqueName: \"kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.571737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.571711 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.628583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.628554 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tlrt\" (UniqueName: \"kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt\") pod \"payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:30.812491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:30.812467 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:31.071993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:31.071908 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc"] Nov 03 00:29:31.083874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:31.083844 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2b087ae-z98lc"] Nov 03 00:29:31.196079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:31.196054 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps"] Nov 03 00:29:31.196526 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:29:31.196503 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9ef3db3_cb62_41ba_b0d0_00251ecceff3.slice/crio-445d44de49bff96af2f9198b971a03c8cc761a564ce6029f691118f81628b4b9 WatchSource:0}: Error finding container 445d44de49bff96af2f9198b971a03c8cc761a564ce6029f691118f81628b4b9: Status 404 returned error can't find the container with id 445d44de49bff96af2f9198b971a03c8cc761a564ce6029f691118f81628b4b9 Nov 03 00:29:31.998016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:31.997976 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9fcf5a-97ef-4f61-8b84-e8709230984c" path="/var/lib/kubelet/pods/0f9fcf5a-97ef-4f61-8b84-e8709230984c/volumes" Nov 03 00:29:32.085721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.085684 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" event={"ID":"d9ef3db3-cb62-41ba-b0d0-00251ecceff3","Type":"ContainerStarted","Data":"3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2"} Nov 03 00:29:32.085902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.085743 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" event={"ID":"d9ef3db3-cb62-41ba-b0d0-00251ecceff3","Type":"ContainerStarted","Data":"445d44de49bff96af2f9198b971a03c8cc761a564ce6029f691118f81628b4b9"} Nov 03 00:29:32.085902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.085785 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:32.201080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.201034 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" podStartSLOduration=2.2010179 podStartE2EDuration="2.2010179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:32.197886377 +0000 UTC m=+1462.343580831" watchObservedRunningTime="2026-11-03 00:29:32.2010179 +0000 UTC m=+1462.346712358" Nov 03 00:29:32.781819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.781789 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl"] Nov 03 00:29:32.786239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.786217 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:32.840869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.840838 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-c521ba\"" Nov 03 00:29:32.846774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.846751 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl"] Nov 03 00:29:32.968960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:32.968922 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-c521ba-dockercfg-c96s8\"" Nov 03 00:29:33.060717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.054072 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdwd2\" (UniqueName: \"kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2\") pod \"payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.065980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.061221 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.167744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.167703 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdwd2\" (UniqueName: \"kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2\") pod \"payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.167901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.167823 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.168514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.168492 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.227110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.227082 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdwd2\" (UniqueName: \"kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2\") pod \"payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.403983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.403916 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:33.790997 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:29:33.790954 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86f3df52_ecd6_479e_9569_583495653896.slice/crio-db4a9f8a082fbf0261ad70e639edcafc206debdb9e39246189a1b36a19394326 WatchSource:0}: Error finding container db4a9f8a082fbf0261ad70e639edcafc206debdb9e39246189a1b36a19394326: Status 404 returned error can't find the container with id db4a9f8a082fbf0261ad70e639edcafc206debdb9e39246189a1b36a19394326 Nov 03 00:29:33.791349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:33.791332 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl"] Nov 03 00:29:34.045446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:34.045366 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf"] Nov 03 00:29:34.094450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:34.094425 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77b65026-tdfpf"] Nov 03 00:29:34.128523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:34.128491 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" event={"ID":"86f3df52-ecd6-479e-9569-583495653896","Type":"ContainerStarted","Data":"7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78"} Nov 03 00:29:34.128726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:34.128544 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" event={"ID":"86f3df52-ecd6-479e-9569-583495653896","Type":"ContainerStarted","Data":"db4a9f8a082fbf0261ad70e639edcafc206debdb9e39246189a1b36a19394326"} Nov 03 00:29:34.128726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:34.128581 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:34.241977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:34.241924 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" podStartSLOduration=2.241911768 podStartE2EDuration="2.241911768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:34.240063744 +0000 UTC m=+1464.385758237" watchObservedRunningTime="2026-11-03 00:29:34.241911768 +0000 UTC m=+1464.387606265" Nov 03 00:29:35.048801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:35.048770 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw"] Nov 03 00:29:35.053205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:35.053181 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9dda34b1-w2bvw"] Nov 03 00:29:35.860115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:35.860076 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw"] Nov 03 00:29:35.975595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:35.975556 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n"] Nov 03 00:29:35.997457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:35.997426 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7cec7c9-baf6-473b-afad-530986351553" path="/var/lib/kubelet/pods/e7cec7c9-baf6-473b-afad-530986351553/volumes" Nov 03 00:29:35.997951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:35.997937 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f45e65-68df-4263-9850-772341980e3e" path="/var/lib/kubelet/pods/f1f45e65-68df-4263-9850-772341980e3e/volumes" Nov 03 00:29:40.861007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:40.860952 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" podUID="2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" containerName="payload-processing" containerID="cri-o://0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884" gracePeriod=25 Nov 03 00:29:40.976712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:40.976605 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" podUID="92e647e4-68b9-4659-967d-8f359de841b9" containerName="payload-pre-processing" containerID="cri-o://6d734fc178d8c5fd33ba9972004293fc219c1f1960e77b50ec65a42bc1b69f5c" gracePeriod=25 Nov 03 00:29:41.158244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.158211 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:29:41.202884 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.202840 5144 generic.go:356] "Generic (PLEG): container finished" podID="2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" containerID="0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884" exitCode=0 Nov 03 00:29:41.203077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.203059 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" event={"ID":"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4","Type":"ContainerDied","Data":"0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884"} Nov 03 00:29:41.203132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.203113 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" event={"ID":"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4","Type":"ContainerDied","Data":"6f90f5e29907936e1e77544615459208b7796e502fd7ccae11f347b2111ef777"} Nov 03 00:29:41.203171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.203139 5144 scope.go:122] "RemoveContainer" containerID="0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884" Nov 03 00:29:41.203304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.203292 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw" Nov 03 00:29:41.206035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.206008 5144 generic.go:356] "Generic (PLEG): container finished" podID="92e647e4-68b9-4659-967d-8f359de841b9" containerID="6d734fc178d8c5fd33ba9972004293fc219c1f1960e77b50ec65a42bc1b69f5c" exitCode=0 Nov 03 00:29:41.206183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.206047 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" event={"ID":"92e647e4-68b9-4659-967d-8f359de841b9","Type":"ContainerDied","Data":"6d734fc178d8c5fd33ba9972004293fc219c1f1960e77b50ec65a42bc1b69f5c"} Nov 03 00:29:41.216485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.216422 5144 scope.go:122] "RemoveContainer" containerID="0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884" Nov 03 00:29:41.216880 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:41.216842 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884\": container with ID starting with 0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884 not found: ID does not exist" containerID="0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884" Nov 03 00:29:41.216971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.216877 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884"} err="failed to get container status \"0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884\": rpc error: code = NotFound desc = could not find container \"0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884\": container with ID starting with 0fd3806c16b437fa4d2a50fa2cb10ea108d655f292c237deebe6cbc5d8d36884 not found: ID does not exist" Nov 03 00:29:41.227587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.227557 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume\") pod \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " Nov 03 00:29:41.227873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.227848 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume" pod "2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" (UID: "2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:41.227986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.227888 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g\" (UniqueName: \"kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g\") pod \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\" (UID: \"2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4\") " Nov 03 00:29:41.228544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.228455 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:41.230191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.230162 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g" pod "2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" (UID: "2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4"). InnerVolumeSpecName "kube-api-access-7fq4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:41.257448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.257429 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:29:41.329358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.329316 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t\" (UniqueName: \"kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t\") pod \"92e647e4-68b9-4659-967d-8f359de841b9\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " Nov 03 00:29:41.329516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.329383 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume\") pod \"92e647e4-68b9-4659-967d-8f359de841b9\" (UID: \"92e647e4-68b9-4659-967d-8f359de841b9\") " Nov 03 00:29:41.329852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.329825 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume" pod "92e647e4-68b9-4659-967d-8f359de841b9" (UID: "92e647e4-68b9-4659-967d-8f359de841b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:41.329996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.329983 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92e647e4-68b9-4659-967d-8f359de841b9-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:41.330043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.330000 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fq4g\" (UniqueName: \"kubernetes.io/projected/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4-kube-api-access-7fq4g\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:41.331583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.331549 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t" pod "92e647e4-68b9-4659-967d-8f359de841b9" (UID: "92e647e4-68b9-4659-967d-8f359de841b9"). InnerVolumeSpecName "kube-api-access-wg96t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:41.431871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.431806 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wg96t\" (UniqueName: \"kubernetes.io/projected/92e647e4-68b9-4659-967d-8f359de841b9-kube-api-access-wg96t\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:41.624916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.624884 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw"] Nov 03 00:29:41.629273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.629251 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-73821c-578dcdf9cf-8t4qw"] Nov 03 00:29:41.997386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:41.997348 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" path="/var/lib/kubelet/pods/2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4/volumes" Nov 03 00:29:42.156112 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:42.156022 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:29:42.216305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:42.216280 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" Nov 03 00:29:42.216459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:42.216284 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n" event={"ID":"92e647e4-68b9-4659-967d-8f359de841b9","Type":"ContainerDied","Data":"84eac5df5c3a8d665854a5851a684d6ea7f079404a9cf96a93c2a1eac33a54d2"} Nov 03 00:29:42.216459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:42.216410 5144 scope.go:122] "RemoveContainer" containerID="6d734fc178d8c5fd33ba9972004293fc219c1f1960e77b50ec65a42bc1b69f5c" Nov 03 00:29:42.380863 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:42.380836 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n"] Nov 03 00:29:42.384915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:42.384893 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-73821c-7985bd48b7-8d59n"] Nov 03 00:29:43.120299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:43.120272 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:29:43.999002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:43.998969 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92e647e4-68b9-4659-967d-8f359de841b9" path="/var/lib/kubelet/pods/92e647e4-68b9-4659-967d-8f359de841b9/volumes" Nov 03 00:29:45.140669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:45.140605 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:29:47.100462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:47.100432 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz"] Nov 03 00:29:47.150246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:47.150217 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c071dcf0-mjrvz"] Nov 03 00:29:48.004947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:48.004913 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2ed52a-865d-4338-8280-4c20f2bc527c" path="/var/lib/kubelet/pods/dc2ed52a-865d-4338-8280-4c20f2bc527c/volumes" Nov 03 00:29:48.879000 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:48.878914 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz"] Nov 03 00:29:48.879499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:48.879126 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" containerID="cri-o://932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741" gracePeriod=30 Nov 03 00:29:54.055031 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.054989 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw"] Nov 03 00:29:54.121953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.121893 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56eea33d-25cpw"] Nov 03 00:29:54.200981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.200959 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223425 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223537 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223589 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223622 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223689 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223752 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.223818 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.224235 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.225669 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.226064 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.226152 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.226192 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo\") pod \"3795cc1f-3ad6-4676-a9b9-e36061737693\" (UID: \"3795cc1f-3ad6-4676-a9b9-e36061737693\") " Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.227254 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3795cc1f-3ad6-4676-a9b9-e36061737693-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.227275 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.227288 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.227668 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:29:54.227989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.227878 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:29:54.231480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.231455 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:29:54.231583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.231543 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:54.231704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.231677 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "kube-api-access-x56sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:54.232902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.232877 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo" pod "3795cc1f-3ad6-4676-a9b9-e36061737693" (UID: "3795cc1f-3ad6-4676-a9b9-e36061737693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:29:54.328585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.328560 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.328585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.328586 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x56sv\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-kube-api-access-x56sv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.328771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.328595 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.328771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.328606 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3795cc1f-3ad6-4676-a9b9-e36061737693-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.328771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.328614 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.328771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.328622 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3795cc1f-3ad6-4676-a9b9-e36061737693-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:54.334837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.334809 5144 generic.go:356] "Generic (PLEG): container finished" podID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerID="932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741" exitCode=0 Nov 03 00:29:54.335022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.334875 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" Nov 03 00:29:54.335022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.334889 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" event={"ID":"3795cc1f-3ad6-4676-a9b9-e36061737693","Type":"ContainerDied","Data":"932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741"} Nov 03 00:29:54.335022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.334948 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz" event={"ID":"3795cc1f-3ad6-4676-a9b9-e36061737693","Type":"ContainerDied","Data":"dd27932f2e919d4a4f4a9d2033c26561270ac8b313d402aea3c49dbcd0ab2115"} Nov 03 00:29:54.335022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.334974 5144 scope.go:122] "RemoveContainer" containerID="932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741" Nov 03 00:29:54.358022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.357997 5144 scope.go:122] "RemoveContainer" containerID="845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51" Nov 03 00:29:54.369061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.369032 5144 scope.go:122] "RemoveContainer" containerID="932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741" Nov 03 00:29:54.369345 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:54.369327 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741\": container with ID starting with 932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741 not found: ID does not exist" containerID="932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741" Nov 03 00:29:54.369444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.369350 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741"} err="failed to get container status \"932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741\": rpc error: code = NotFound desc = could not find container \"932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741\": container with ID starting with 932fe8f90da6788c9c394dff8d987d2da74084c7f42e195e7b59b20f57f1e741 not found: ID does not exist" Nov 03 00:29:54.369444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.369364 5144 scope.go:122] "RemoveContainer" containerID="845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51" Nov 03 00:29:54.369625 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:29:54.369604 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51\": container with ID starting with 845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51 not found: ID does not exist" containerID="845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51" Nov 03 00:29:54.369625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.369645 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51"} err="failed to get container status \"845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51\": rpc error: code = NotFound desc = could not find container \"845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51\": container with ID starting with 845f3906117461148f0aad592916be2cfd59f1a63420e802c112a59a4f736a51 not found: ID does not exist" Nov 03 00:29:54.501032 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.500976 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz"] Nov 03 00:29:54.505015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:54.504995 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a9404c-openshift-default-5466bc4f7f-wrjqz"] Nov 03 00:29:55.056534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.056502 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2"] Nov 03 00:29:55.057220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057194 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" Nov 03 00:29:55.057220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057211 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" Nov 03 00:29:55.057303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057235 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" containerName="payload-processing" Nov 03 00:29:55.057303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057241 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" containerName="payload-processing" Nov 03 00:29:55.057303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057261 5144 cpu_manager.go:414] "containerMap: removing container" podUID="92e647e4-68b9-4659-967d-8f359de841b9" containerName="payload-pre-processing" Nov 03 00:29:55.057303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057266 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92e647e4-68b9-4659-967d-8f359de841b9" containerName="payload-pre-processing" Nov 03 00:29:55.057426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057381 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" Nov 03 00:29:55.057426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057392 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2621f2a0-7c0a-42cd-b3ba-2bf8493a78d4" containerName="payload-processing" Nov 03 00:29:55.057426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.057402 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="92e647e4-68b9-4659-967d-8f359de841b9" containerName="payload-pre-processing" Nov 03 00:29:55.060530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.060515 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.117394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.117370 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:29:55.117808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.117400 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:29:55.117808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.117441 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2"] Nov 03 00:29:55.139184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.139159 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.139308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.139201 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-875xd\" (UniqueName: \"kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.139452 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.139428 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.240871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.240847 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.241027 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.240881 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-875xd\" (UniqueName: \"kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.241027 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.241011 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.241651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.241586 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.243600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.243581 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.297810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.297779 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-875xd\" (UniqueName: \"kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd\") pod \"maas-api-revoke-sub-294ca089-d9qn2\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.373674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.373572 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:55.764453 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:29:55.764420 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c0bb11_ddfc_4022_bed0_ff0ebaf09393.slice/crio-ae7a53d17782ed4f57c862fd4dd598907ab87a631eeb0e0a0f8fbab5f7df4d28 WatchSource:0}: Error finding container ae7a53d17782ed4f57c862fd4dd598907ab87a631eeb0e0a0f8fbab5f7df4d28: Status 404 returned error can't find the container with id ae7a53d17782ed4f57c862fd4dd598907ab87a631eeb0e0a0f8fbab5f7df4d28 Nov 03 00:29:55.767406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.767378 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2"] Nov 03 00:29:55.998003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.997974 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30df45df-f5e5-4ade-818d-d9acede7795a" path="/var/lib/kubelet/pods/30df45df-f5e5-4ade-818d-d9acede7795a/volumes" Nov 03 00:29:55.998461 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:55.998446 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" path="/var/lib/kubelet/pods/3795cc1f-3ad6-4676-a9b9-e36061737693/volumes" Nov 03 00:29:56.356701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:56.356605 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" event={"ID":"11c0bb11-ddfc-4022-bed0-ff0ebaf09393","Type":"ContainerStarted","Data":"10ad300b0c1d719f3e847fe324bfd5fce1940ace4e65632f94477ac8bf08b8c5"} Nov 03 00:29:56.356701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:56.356675 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" event={"ID":"11c0bb11-ddfc-4022-bed0-ff0ebaf09393","Type":"ContainerStarted","Data":"ae7a53d17782ed4f57c862fd4dd598907ab87a631eeb0e0a0f8fbab5f7df4d28"} Nov 03 00:29:56.384169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:56.384142 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l"] Nov 03 00:29:56.473613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:56.473568 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" podStartSLOduration=1.473556406 podStartE2EDuration="1.473556406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:56.469461672 +0000 UTC m=+1486.615156146" watchObservedRunningTime="2026-11-03 00:29:56.473556406 +0000 UTC m=+1486.619250861" Nov 03 00:29:56.497780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:56.497753 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq"] Nov 03 00:29:57.366753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:57.366719 5144 generic.go:356] "Generic (PLEG): container finished" podID="11c0bb11-ddfc-4022-bed0-ff0ebaf09393" containerID="10ad300b0c1d719f3e847fe324bfd5fce1940ace4e65632f94477ac8bf08b8c5" exitCode=0 Nov 03 00:29:57.367136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:57.366806 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" event={"ID":"11c0bb11-ddfc-4022-bed0-ff0ebaf09393","Type":"ContainerDied","Data":"10ad300b0c1d719f3e847fe324bfd5fce1940ace4e65632f94477ac8bf08b8c5"} Nov 03 00:29:58.053876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.053843 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw"] Nov 03 00:29:58.103903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.103871 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b29c2fc-khjcw"] Nov 03 00:29:58.559725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.559700 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:58.610547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.610522 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca\") pod \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " Nov 03 00:29:58.610707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.610688 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd\" (UniqueName: \"kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd\") pod \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " Nov 03 00:29:58.610784 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.610768 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca\") pod \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\" (UID: \"11c0bb11-ddfc-4022-bed0-ff0ebaf09393\") " Nov 03 00:29:58.610969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.610943 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca" pod "11c0bb11-ddfc-4022-bed0-ff0ebaf09393" (UID: "11c0bb11-ddfc-4022-bed0-ff0ebaf09393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:29:58.611502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.611414 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:58.612918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.612891 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd" pod "11c0bb11-ddfc-4022-bed0-ff0ebaf09393" (UID: "11c0bb11-ddfc-4022-bed0-ff0ebaf09393"). InnerVolumeSpecName "kube-api-access-875xd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:29:58.613324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.613305 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca" pod "11c0bb11-ddfc-4022-bed0-ff0ebaf09393" (UID: "11c0bb11-ddfc-4022-bed0-ff0ebaf09393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:29:58.714011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.713925 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:58.714011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:58.713976 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-875xd\" (UniqueName: \"kubernetes.io/projected/11c0bb11-ddfc-4022-bed0-ff0ebaf09393-kube-api-access-875xd\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:29:59.387102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:59.387074 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" Nov 03 00:29:59.387273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:59.387115 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2" event={"ID":"11c0bb11-ddfc-4022-bed0-ff0ebaf09393","Type":"ContainerDied","Data":"ae7a53d17782ed4f57c862fd4dd598907ab87a631eeb0e0a0f8fbab5f7df4d28"} Nov 03 00:29:59.387273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:59.387160 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae7a53d17782ed4f57c862fd4dd598907ab87a631eeb0e0a0f8fbab5f7df4d28" Nov 03 00:29:59.997311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:29:59.997277 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9661b07-ed83-4ec4-9948-762b602dff13" path="/var/lib/kubelet/pods/f9661b07-ed83-4ec4-9948-762b602dff13/volumes" Nov 03 00:30:00.168450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.168421 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj"] Nov 03 00:30:00.169508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.169488 5144 cpu_manager.go:414] "containerMap: removing container" podUID="11c0bb11-ddfc-4022-bed0-ff0ebaf09393" containerName="revoke-keys" Nov 03 00:30:00.169574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.169513 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11c0bb11-ddfc-4022-bed0-ff0ebaf09393" containerName="revoke-keys" Nov 03 00:30:00.169574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.169544 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" Nov 03 00:30:00.169574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.169551 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" Nov 03 00:30:00.169768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.169757 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="11c0bb11-ddfc-4022-bed0-ff0ebaf09393" containerName="revoke-keys" Nov 03 00:30:00.169799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.169774 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3795cc1f-3ad6-4676-a9b9-e36061737693" containerName="istio-proxy" Nov 03 00:30:00.173871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.173850 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.174042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.174020 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf"] Nov 03 00:30:00.177201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.177179 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.177761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.177744 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht"] Nov 03 00:30:00.180649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.180613 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.217879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.217860 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz"] Nov 03 00:30:00.221041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.221026 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.221514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.221495 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp"] Nov 03 00:30:00.224613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.224597 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.225167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.225146 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz"] Nov 03 00:30:00.225739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.225722 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:30:00.228792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.228773 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.229147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.229118 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs"] Nov 03 00:30:00.232097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.232079 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.241761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.241733 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfs79\" (UniqueName: \"kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.241924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.241903 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv9n6\" (UniqueName: \"kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.242166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.241973 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.242285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.242179 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7rsw\" (UniqueName: \"kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw\") pod \"maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.242285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.242231 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.242415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.242295 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.270462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.270441 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj"] Nov 03 00:30:00.277222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.277188 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf"] Nov 03 00:30:00.325852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.325824 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp"] Nov 03 00:30:00.328457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.328435 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht"] Nov 03 00:30:00.334919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.334895 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz"] Nov 03 00:30:00.341435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.341412 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs"] Nov 03 00:30:00.343710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.343681 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.343822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.343724 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6xdx\" (UniqueName: \"kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx\") pod \"maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.343895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.343819 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv9n6\" (UniqueName: \"kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.343957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.343922 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqsqd\" (UniqueName: \"kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd\") pod \"maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.344013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.343973 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.344074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344024 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.344127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344075 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7rsw\" (UniqueName: \"kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw\") pod \"maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.344186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344126 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.344241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344226 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.344297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344277 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pnnp\" (UniqueName: \"kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.344383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344363 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.344457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344441 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfs79\" (UniqueName: \"kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.344521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344477 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrkbq\" (UniqueName: \"kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.344581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344518 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.344857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344834 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.344928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.344875 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.345366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.345342 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.376057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.375676 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz"] Nov 03 00:30:00.401220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.401176 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfs79\" (UniqueName: \"kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.401739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.401710 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv9n6\" (UniqueName: \"kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.401853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.401787 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7rsw\" (UniqueName: \"kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw\") pod \"maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.445517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445489 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.445661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445577 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.445661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445608 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6xdx\" (UniqueName: \"kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx\") pod \"maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.445779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445713 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqsqd\" (UniqueName: \"kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd\") pod \"maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.445779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445760 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.445924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445902 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pnnp\" (UniqueName: \"kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.446013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.445990 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.446096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.446070 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrkbq\" (UniqueName: \"kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.446336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.446301 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.446416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.446303 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.446531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.446513 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.446679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.446659 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.489230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.489209 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:00.496752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.496729 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:00.503278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.502959 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:00.503278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.503205 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrkbq\" (UniqueName: \"kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.503278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.503218 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pnnp\" (UniqueName: \"kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.503471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.503299 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqsqd\" (UniqueName: \"kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd\") pod \"maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.503789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.503770 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6xdx\" (UniqueName: \"kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx\") pod \"maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.542075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.541946 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:00.542787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.542760 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:00.552928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.552758 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:00.566533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.566445 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:00.916941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:00.916915 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht"] Nov 03 00:30:00.917415 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:00.917370 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138fa9ce_6c3b_4115_9104_993411563d70.slice/crio-4928cf343d254d0268db1b979af663b4aee67951cb279a57391757a98b58f563 WatchSource:0}: Error finding container 4928cf343d254d0268db1b979af663b4aee67951cb279a57391757a98b58f563: Status 404 returned error can't find the container with id 4928cf343d254d0268db1b979af663b4aee67951cb279a57391757a98b58f563 Nov 03 00:30:01.024689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.024663 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz"] Nov 03 00:30:01.025790 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:01.025764 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1241378f_7ae1_468b_98cf_2b69455d4a63.slice/crio-74ca0f319ab8374745784016a5e5f7dbefd828a6c1260a83985b871be502828d WatchSource:0}: Error finding container 74ca0f319ab8374745784016a5e5f7dbefd828a6c1260a83985b871be502828d: Status 404 returned error can't find the container with id 74ca0f319ab8374745784016a5e5f7dbefd828a6c1260a83985b871be502828d Nov 03 00:30:01.026320 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:01.026299 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7f037a_ae59_4ef2_8580_824e5b2cecf2.slice/crio-bcbeb5b3e7e054a420cbf1340d31fd131b8d529b437a95fcd3bd8922b890e52b WatchSource:0}: Error finding container bcbeb5b3e7e054a420cbf1340d31fd131b8d529b437a95fcd3bd8922b890e52b: Status 404 returned error can't find the container with id bcbeb5b3e7e054a420cbf1340d31fd131b8d529b437a95fcd3bd8922b890e52b Nov 03 00:30:01.029816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.029796 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs"] Nov 03 00:30:01.087496 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.087408 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp"] Nov 03 00:30:01.094512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.094474 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz"] Nov 03 00:30:01.098569 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:01.097795 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3357bb_1ff9_4e8f_bc9b_4c23f48a7a80.slice/crio-c70170d4224d5155d443dc5e673b161eceb3f920e21733e8bed7347254736be0 WatchSource:0}: Error finding container c70170d4224d5155d443dc5e673b161eceb3f920e21733e8bed7347254736be0: Status 404 returned error can't find the container with id c70170d4224d5155d443dc5e673b161eceb3f920e21733e8bed7347254736be0 Nov 03 00:30:01.106617 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:01.103003 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffd5a64_ffbd_4c8a_9d85_7626c6c226a6.slice/crio-c27c045b4ab7d06b50563644d0d73d420258229b6d685bf49f86000ceff78117 WatchSource:0}: Error finding container c27c045b4ab7d06b50563644d0d73d420258229b6d685bf49f86000ceff78117: Status 404 returned error can't find the container with id c27c045b4ab7d06b50563644d0d73d420258229b6d685bf49f86000ceff78117 Nov 03 00:30:01.111585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.111557 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf"] Nov 03 00:30:01.134282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.134248 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj"] Nov 03 00:30:01.218874 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:01.218841 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fdd47aa_8edd_4563_b1a6_6a6f6977e893.slice/crio-83dc158631b83d0c022fa2755635ca47b83b5928c597c765b66f5e0c78f211ef WatchSource:0}: Error finding container 83dc158631b83d0c022fa2755635ca47b83b5928c597c765b66f5e0c78f211ef: Status 404 returned error can't find the container with id 83dc158631b83d0c022fa2755635ca47b83b5928c597c765b66f5e0c78f211ef Nov 03 00:30:01.220796 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:01.220771 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbf00a1f_da54_4267_8e1f_6b94c90b0b8a.slice/crio-fe1f0cd9d4b2d7ce8efa85f32b27b726d01d0ecf79d0751ccd9a740909a68a3a WatchSource:0}: Error finding container fe1f0cd9d4b2d7ce8efa85f32b27b726d01d0ecf79d0751ccd9a740909a68a3a: Status 404 returned error can't find the container with id fe1f0cd9d4b2d7ce8efa85f32b27b726d01d0ecf79d0751ccd9a740909a68a3a Nov 03 00:30:01.385403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.385368 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" podUID="de9530be-8f13-47e4-a5b9-d18c645d243f" containerName="payload-processing" containerID="cri-o://386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356" gracePeriod=25 Nov 03 00:30:01.406193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.406155 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerStarted","Data":"c70170d4224d5155d443dc5e673b161eceb3f920e21733e8bed7347254736be0"} Nov 03 00:30:01.407548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.407525 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerStarted","Data":"bcbeb5b3e7e054a420cbf1340d31fd131b8d529b437a95fcd3bd8922b890e52b"} Nov 03 00:30:01.408899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.408880 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerStarted","Data":"4928cf343d254d0268db1b979af663b4aee67951cb279a57391757a98b58f563"} Nov 03 00:30:01.410188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.410161 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" event={"ID":"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a","Type":"ContainerStarted","Data":"fe1f0cd9d4b2d7ce8efa85f32b27b726d01d0ecf79d0751ccd9a740909a68a3a"} Nov 03 00:30:01.411490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.411467 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerStarted","Data":"74ca0f319ab8374745784016a5e5f7dbefd828a6c1260a83985b871be502828d"} Nov 03 00:30:01.412777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.412759 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerStarted","Data":"83dc158631b83d0c022fa2755635ca47b83b5928c597c765b66f5e0c78f211ef"} Nov 03 00:30:01.414281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.414258 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerStarted","Data":"c27c045b4ab7d06b50563644d0d73d420258229b6d685bf49f86000ceff78117"} Nov 03 00:30:01.498294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.498254 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" podUID="a7235f6d-4796-4b2c-af54-0f13aab7bbea" containerName="payload-pre-processing" containerID="cri-o://8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c" gracePeriod=25 Nov 03 00:30:01.831824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.831799 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:30:01.974368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.974337 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume\") pod \"de9530be-8f13-47e4-a5b9-d18c645d243f\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " Nov 03 00:30:01.974535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.974503 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5\" (UniqueName: \"kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5\") pod \"de9530be-8f13-47e4-a5b9-d18c645d243f\" (UID: \"de9530be-8f13-47e4-a5b9-d18c645d243f\") " Nov 03 00:30:01.974828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.974801 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume" pod "de9530be-8f13-47e4-a5b9-d18c645d243f" (UID: "de9530be-8f13-47e4-a5b9-d18c645d243f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:01.982613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:01.982594 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:30:02.018240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.018198 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5" pod "de9530be-8f13-47e4-a5b9-d18c645d243f" (UID: "de9530be-8f13-47e4-a5b9-d18c645d243f"). InnerVolumeSpecName "kube-api-access-b28l5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:02.075890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.075859 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb\" (UniqueName: \"kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb\") pod \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " Nov 03 00:30:02.076281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.075970 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume\") pod \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\" (UID: \"a7235f6d-4796-4b2c-af54-0f13aab7bbea\") " Nov 03 00:30:02.076402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.076369 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume" pod "a7235f6d-4796-4b2c-af54-0f13aab7bbea" (UID: "a7235f6d-4796-4b2c-af54-0f13aab7bbea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:02.076943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.076921 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de9530be-8f13-47e4-a5b9-d18c645d243f-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:02.077041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.076946 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7235f6d-4796-4b2c-af54-0f13aab7bbea-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:02.077041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.076961 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b28l5\" (UniqueName: \"kubernetes.io/projected/de9530be-8f13-47e4-a5b9-d18c645d243f-kube-api-access-b28l5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:02.078448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.078426 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb" pod "a7235f6d-4796-4b2c-af54-0f13aab7bbea" (UID: "a7235f6d-4796-4b2c-af54-0f13aab7bbea"). InnerVolumeSpecName "kube-api-access-jscpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:02.178190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.178163 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jscpb\" (UniqueName: \"kubernetes.io/projected/a7235f6d-4796-4b2c-af54-0f13aab7bbea-kube-api-access-jscpb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:02.444738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.444696 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerStarted","Data":"b6f6356542502b5598e3e0a3f7b2caa17ca9a6e0bd2af5ff8a6ad4ba2c7ac339"} Nov 03 00:30:02.446985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.446956 5144 generic.go:356] "Generic (PLEG): container finished" podID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerID="f95df1873549f53f17ed2b226d02d4a86c87fe8256c8a4a2011a23b0cc07eed4" exitCode=60 Nov 03 00:30:02.447116 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.447047 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerDied","Data":"f95df1873549f53f17ed2b226d02d4a86c87fe8256c8a4a2011a23b0cc07eed4"} Nov 03 00:30:02.447287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.447269 5144 scope.go:122] "RemoveContainer" containerID="f95df1873549f53f17ed2b226d02d4a86c87fe8256c8a4a2011a23b0cc07eed4" Nov 03 00:30:02.449488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.449108 5144 generic.go:356] "Generic (PLEG): container finished" podID="138fa9ce-6c3b-4115-9104-993411563d70" containerID="96007079af9eb1e2f2b2c100d19f1e637be79bbfb87175f206bc3a112e19fe40" exitCode=60 Nov 03 00:30:02.449488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.449292 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerDied","Data":"96007079af9eb1e2f2b2c100d19f1e637be79bbfb87175f206bc3a112e19fe40"} Nov 03 00:30:02.449612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.449544 5144 scope.go:122] "RemoveContainer" containerID="96007079af9eb1e2f2b2c100d19f1e637be79bbfb87175f206bc3a112e19fe40" Nov 03 00:30:02.451570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.451548 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" event={"ID":"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a","Type":"ContainerStarted","Data":"1821b5ed6c62c1ae46576db31b085ef26d60bcdfd46f90ae85f8f8cbfbc044e5"} Nov 03 00:30:02.453346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.453327 5144 generic.go:356] "Generic (PLEG): container finished" podID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerID="961fe2659f5fc142a9d6d5261a7e15b322a6b3339b20bad1c0aeb6bd8265bc22" exitCode=60 Nov 03 00:30:02.453419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.453392 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerDied","Data":"961fe2659f5fc142a9d6d5261a7e15b322a6b3339b20bad1c0aeb6bd8265bc22"} Nov 03 00:30:02.453577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.453564 5144 scope.go:122] "RemoveContainer" containerID="961fe2659f5fc142a9d6d5261a7e15b322a6b3339b20bad1c0aeb6bd8265bc22" Nov 03 00:30:02.454997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.454974 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerStarted","Data":"6a606b2ec922bc3f3b3fd17e6ec6b1bc21d0ae808a1f7c9fb0e64df9de83aead"} Nov 03 00:30:02.462859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.462839 5144 generic.go:356] "Generic (PLEG): container finished" podID="de9530be-8f13-47e4-a5b9-d18c645d243f" containerID="386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356" exitCode=0 Nov 03 00:30:02.462961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.462862 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" event={"ID":"de9530be-8f13-47e4-a5b9-d18c645d243f","Type":"ContainerDied","Data":"386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356"} Nov 03 00:30:02.462961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.462890 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" Nov 03 00:30:02.462961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.462908 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l" event={"ID":"de9530be-8f13-47e4-a5b9-d18c645d243f","Type":"ContainerDied","Data":"6cfcd735182632f91099374e0d3ef7e96f422d4b24f51381d4958e51e8ecc7a8"} Nov 03 00:30:02.462961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.462938 5144 scope.go:122] "RemoveContainer" containerID="386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356" Nov 03 00:30:02.465370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.465353 5144 generic.go:356] "Generic (PLEG): container finished" podID="a7235f6d-4796-4b2c-af54-0f13aab7bbea" containerID="8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c" exitCode=0 Nov 03 00:30:02.465689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.465531 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" Nov 03 00:30:02.465780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.465544 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" event={"ID":"a7235f6d-4796-4b2c-af54-0f13aab7bbea","Type":"ContainerDied","Data":"8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c"} Nov 03 00:30:02.465830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.465799 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq" event={"ID":"a7235f6d-4796-4b2c-af54-0f13aab7bbea","Type":"ContainerDied","Data":"73b5e2d047c908de27722473cd7cd0650e2d1b9c8a54471f561b6aeb8e538225"} Nov 03 00:30:02.470974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.470954 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerStarted","Data":"8bbcf2f8e961477df9c54567804efc2b509bdff70ba1357e8ab6b5efaf1f1ad9"} Nov 03 00:30:02.481558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.481536 5144 scope.go:122] "RemoveContainer" containerID="386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356" Nov 03 00:30:02.481902 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:02.481837 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356\": container with ID starting with 386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356 not found: ID does not exist" containerID="386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356" Nov 03 00:30:02.481902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.481861 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356"} err="failed to get container status \"386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356\": rpc error: code = NotFound desc = could not find container \"386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356\": container with ID starting with 386d8db4c730398facf9d14f210bd67bb55b81c9aec2451082b2611a7d96d356 not found: ID does not exist" Nov 03 00:30:02.481902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.481877 5144 scope.go:122] "RemoveContainer" containerID="8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c" Nov 03 00:30:02.526883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.526861 5144 scope.go:122] "RemoveContainer" containerID="8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c" Nov 03 00:30:02.527193 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:02.527172 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c\": container with ID starting with 8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c not found: ID does not exist" containerID="8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c" Nov 03 00:30:02.527259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.527201 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c"} err="failed to get container status \"8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c\": rpc error: code = NotFound desc = could not find container \"8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c\": container with ID starting with 8fb5f31aaa2e32617e607efd3e55d40b6bec19a9f51f8942db2a66672b0ffc2c not found: ID does not exist" Nov 03 00:30:02.560083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.560039 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" podStartSLOduration=2.560026281 podStartE2EDuration="2.560026281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:02.556251755 +0000 UTC m=+1492.701946203" watchObservedRunningTime="2026-11-03 00:30:02.560026281 +0000 UTC m=+1492.705720733" Nov 03 00:30:02.668618 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.668569 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" podStartSLOduration=2.668555323 podStartE2EDuration="2.668555323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:02.668107308 +0000 UTC m=+1492.813801762" watchObservedRunningTime="2026-11-03 00:30:02.668555323 +0000 UTC m=+1492.814249779" Nov 03 00:30:02.896170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:02.896120 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" podStartSLOduration=2.89610552 podStartE2EDuration="2.89610552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:02.89587033 +0000 UTC m=+1493.041564813" watchObservedRunningTime="2026-11-03 00:30:02.89610552 +0000 UTC m=+1493.041800006" Nov 03 00:30:03.009128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.009095 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l"] Nov 03 00:30:03.014288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.014264 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e671ea-569dccd987-qhh5l"] Nov 03 00:30:03.349672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.349609 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" podStartSLOduration=3.349595928 podStartE2EDuration="3.349595928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:03.345625831 +0000 UTC m=+1493.491320286" watchObservedRunningTime="2026-11-03 00:30:03.349595928 +0000 UTC m=+1493.495290383" Nov 03 00:30:03.461779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.461747 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq"] Nov 03 00:30:03.467961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.467936 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e671ea-7cd978bf59-tvvmq"] Nov 03 00:30:03.481143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.481111 5144 generic.go:356] "Generic (PLEG): container finished" podID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerID="8bbcf2f8e961477df9c54567804efc2b509bdff70ba1357e8ab6b5efaf1f1ad9" exitCode=60 Nov 03 00:30:03.481267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.481235 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerDied","Data":"8bbcf2f8e961477df9c54567804efc2b509bdff70ba1357e8ab6b5efaf1f1ad9"} Nov 03 00:30:03.481508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.481493 5144 scope.go:122] "RemoveContainer" containerID="8bbcf2f8e961477df9c54567804efc2b509bdff70ba1357e8ab6b5efaf1f1ad9" Nov 03 00:30:03.483029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.483011 5144 generic.go:356] "Generic (PLEG): container finished" podID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerID="b6f6356542502b5598e3e0a3f7b2caa17ca9a6e0bd2af5ff8a6ad4ba2c7ac339" exitCode=60 Nov 03 00:30:03.483135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.483073 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerDied","Data":"b6f6356542502b5598e3e0a3f7b2caa17ca9a6e0bd2af5ff8a6ad4ba2c7ac339"} Nov 03 00:30:03.483331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.483315 5144 scope.go:122] "RemoveContainer" containerID="b6f6356542502b5598e3e0a3f7b2caa17ca9a6e0bd2af5ff8a6ad4ba2c7ac339" Nov 03 00:30:03.485238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.485215 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerStarted","Data":"d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613"} Nov 03 00:30:03.487395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.487375 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerStarted","Data":"d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a"} Nov 03 00:30:03.488995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.488975 5144 generic.go:356] "Generic (PLEG): container finished" podID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerID="1821b5ed6c62c1ae46576db31b085ef26d60bcdfd46f90ae85f8f8cbfbc044e5" exitCode=60 Nov 03 00:30:03.489103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.489056 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" event={"ID":"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a","Type":"ContainerDied","Data":"1821b5ed6c62c1ae46576db31b085ef26d60bcdfd46f90ae85f8f8cbfbc044e5"} Nov 03 00:30:03.489228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.489214 5144 scope.go:122] "RemoveContainer" containerID="1821b5ed6c62c1ae46576db31b085ef26d60bcdfd46f90ae85f8f8cbfbc044e5" Nov 03 00:30:03.491038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.491016 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerStarted","Data":"012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8"} Nov 03 00:30:03.492467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.492445 5144 generic.go:356] "Generic (PLEG): container finished" podID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerID="6a606b2ec922bc3f3b3fd17e6ec6b1bc21d0ae808a1f7c9fb0e64df9de83aead" exitCode=60 Nov 03 00:30:03.492564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.492524 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerDied","Data":"6a606b2ec922bc3f3b3fd17e6ec6b1bc21d0ae808a1f7c9fb0e64df9de83aead"} Nov 03 00:30:03.492740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.492724 5144 scope.go:122] "RemoveContainer" containerID="6a606b2ec922bc3f3b3fd17e6ec6b1bc21d0ae808a1f7c9fb0e64df9de83aead" Nov 03 00:30:03.935341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.935287 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" podStartSLOduration=3.93527073 podStartE2EDuration="3.93527073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:03.928741054 +0000 UTC m=+1494.074435529" watchObservedRunningTime="2026-11-03 00:30:03.93527073 +0000 UTC m=+1494.080965187" Nov 03 00:30:03.998104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.998076 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7235f6d-4796-4b2c-af54-0f13aab7bbea" path="/var/lib/kubelet/pods/a7235f6d-4796-4b2c-af54-0f13aab7bbea/volumes" Nov 03 00:30:03.998593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:03.998578 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de9530be-8f13-47e4-a5b9-d18c645d243f" path="/var/lib/kubelet/pods/de9530be-8f13-47e4-a5b9-d18c645d243f/volumes" Nov 03 00:30:04.048624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.048582 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" podStartSLOduration=4.048569872 podStartE2EDuration="4.048569872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:04.043003912 +0000 UTC m=+1494.188698366" watchObservedRunningTime="2026-11-03 00:30:04.048569872 +0000 UTC m=+1494.194264351" Nov 03 00:30:04.058027 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.058002 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj"] Nov 03 00:30:04.062755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.062732 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-303bb17f-xq6vj"] Nov 03 00:30:04.505125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.505078 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" event={"ID":"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a","Type":"ContainerStarted","Data":"367acb98f496d41c2c4ecaf00c73645cd5ca635c9619a23bb97ad9b35779f928"} Nov 03 00:30:04.507149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.507120 5144 generic.go:356] "Generic (PLEG): container finished" podID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerID="012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8" exitCode=60 Nov 03 00:30:04.507293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.507207 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerDied","Data":"012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8"} Nov 03 00:30:04.507293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.507258 5144 scope.go:122] "RemoveContainer" containerID="961fe2659f5fc142a9d6d5261a7e15b322a6b3339b20bad1c0aeb6bd8265bc22" Nov 03 00:30:04.507434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.507416 5144 scope.go:122] "RemoveContainer" containerID="012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8" Nov 03 00:30:04.507808 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:04.507787 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz_odh-ai-gateway-infra(1241378f-7ae1-468b-98cf-2b69455d4a63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" Nov 03 00:30:04.509337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.509314 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerStarted","Data":"5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a"} Nov 03 00:30:04.511343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.511321 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerStarted","Data":"a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509"} Nov 03 00:30:04.513274 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.513249 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerStarted","Data":"5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a"} Nov 03 00:30:04.516449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.516419 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerDied","Data":"d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613"} Nov 03 00:30:04.516551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.516499 5144 generic.go:356] "Generic (PLEG): container finished" podID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerID="d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613" exitCode=60 Nov 03 00:30:04.517021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.516869 5144 scope.go:122] "RemoveContainer" containerID="d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613" Nov 03 00:30:04.517313 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:04.517290 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs_odh-ai-gateway-infra(3a7f037a-ae59-4ef2-8580-824e5b2cecf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" Nov 03 00:30:04.522656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.519220 5144 generic.go:356] "Generic (PLEG): container finished" podID="138fa9ce-6c3b-4115-9104-993411563d70" containerID="d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a" exitCode=60 Nov 03 00:30:04.522656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.519257 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerDied","Data":"d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a"} Nov 03 00:30:04.522656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.519504 5144 scope.go:122] "RemoveContainer" containerID="d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a" Nov 03 00:30:04.522656 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:04.519806 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht_odh-ai-gateway-infra(138fa9ce-6c3b-4115-9104-993411563d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" podUID="138fa9ce-6c3b-4115-9104-993411563d70" Nov 03 00:30:04.524617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.524596 5144 scope.go:122] "RemoveContainer" containerID="f95df1873549f53f17ed2b226d02d4a86c87fe8256c8a4a2011a23b0cc07eed4" Nov 03 00:30:04.538337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.538312 5144 scope.go:122] "RemoveContainer" containerID="96007079af9eb1e2f2b2c100d19f1e637be79bbfb87175f206bc3a112e19fe40" Nov 03 00:30:04.623255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:04.623201 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" podStartSLOduration=4.623182138 podStartE2EDuration="4.623182138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:04.269844775 +0000 UTC m=+1494.415539258" watchObservedRunningTime="2026-11-03 00:30:04.623182138 +0000 UTC m=+1494.768876595" Nov 03 00:30:05.533076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.533048 5144 scope.go:122] "RemoveContainer" containerID="012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8" Nov 03 00:30:05.533473 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.533346 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz_odh-ai-gateway-infra(1241378f-7ae1-468b-98cf-2b69455d4a63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" Nov 03 00:30:05.534975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.534947 5144 generic.go:356] "Generic (PLEG): container finished" podID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerID="5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a" exitCode=60 Nov 03 00:30:05.535113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.535024 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerDied","Data":"5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a"} Nov 03 00:30:05.535113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.535077 5144 scope.go:122] "RemoveContainer" containerID="6a606b2ec922bc3f3b3fd17e6ec6b1bc21d0ae808a1f7c9fb0e64df9de83aead" Nov 03 00:30:05.535276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.535259 5144 scope.go:122] "RemoveContainer" containerID="5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a" Nov 03 00:30:05.535536 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.535516 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf_odh-ai-gateway-infra(3fdd47aa-8edd-4563-b1a6-6a6f6977e893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" Nov 03 00:30:05.537324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.537302 5144 generic.go:356] "Generic (PLEG): container finished" podID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerID="a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509" exitCode=60 Nov 03 00:30:05.537420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.537382 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerDied","Data":"a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509"} Nov 03 00:30:05.537558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.537544 5144 scope.go:122] "RemoveContainer" containerID="a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509" Nov 03 00:30:05.537797 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.537775 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp_odh-ai-gateway-infra(1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" Nov 03 00:30:05.539599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.539577 5144 generic.go:356] "Generic (PLEG): container finished" podID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerID="5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a" exitCode=60 Nov 03 00:30:05.539738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.539708 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerDied","Data":"5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a"} Nov 03 00:30:05.539913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.539894 5144 scope.go:122] "RemoveContainer" containerID="5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a" Nov 03 00:30:05.540193 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.540170 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz_odh-ai-gateway-infra(9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" Nov 03 00:30:05.541977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.541958 5144 scope.go:122] "RemoveContainer" containerID="d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613" Nov 03 00:30:05.542236 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.542210 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs_odh-ai-gateway-infra(3a7f037a-ae59-4ef2-8580-824e5b2cecf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" Nov 03 00:30:05.544042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.544025 5144 scope.go:122] "RemoveContainer" containerID="d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a" Nov 03 00:30:05.544402 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.544383 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht_odh-ai-gateway-infra(138fa9ce-6c3b-4115-9104-993411563d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" podUID="138fa9ce-6c3b-4115-9104-993411563d70" Nov 03 00:30:05.546814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.546791 5144 generic.go:356] "Generic (PLEG): container finished" podID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerID="367acb98f496d41c2c4ecaf00c73645cd5ca635c9619a23bb97ad9b35779f928" exitCode=60 Nov 03 00:30:05.546919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.546838 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" event={"ID":"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a","Type":"ContainerDied","Data":"367acb98f496d41c2c4ecaf00c73645cd5ca635c9619a23bb97ad9b35779f928"} Nov 03 00:30:05.547121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.547106 5144 scope.go:122] "RemoveContainer" containerID="367acb98f496d41c2c4ecaf00c73645cd5ca635c9619a23bb97ad9b35779f928" Nov 03 00:30:05.547373 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:05.547353 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj_odh-ai-gateway-infra(fbf00a1f-da54-4267-8e1f-6b94c90b0b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" Nov 03 00:30:05.551475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.551459 5144 scope.go:122] "RemoveContainer" containerID="8bbcf2f8e961477df9c54567804efc2b509bdff70ba1357e8ab6b5efaf1f1ad9" Nov 03 00:30:05.574769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.574751 5144 scope.go:122] "RemoveContainer" containerID="b6f6356542502b5598e3e0a3f7b2caa17ca9a6e0bd2af5ff8a6ad4ba2c7ac339" Nov 03 00:30:05.634812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.634782 5144 scope.go:122] "RemoveContainer" containerID="1821b5ed6c62c1ae46576db31b085ef26d60bcdfd46f90ae85f8f8cbfbc044e5" Nov 03 00:30:05.763584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:05.763555 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj"] Nov 03 00:30:06.005240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:06.005201 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0257d36e-5033-467c-aad4-ce1248609d00" path="/var/lib/kubelet/pods/0257d36e-5033-467c-aad4-ce1248609d00/volumes" Nov 03 00:30:06.073259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:06.073216 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm"] Nov 03 00:30:06.559687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:06.558713 5144 scope.go:122] "RemoveContainer" containerID="a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509" Nov 03 00:30:06.559687 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:06.559298 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp_odh-ai-gateway-infra(1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" Nov 03 00:30:06.563686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:06.560893 5144 scope.go:122] "RemoveContainer" containerID="5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a" Nov 03 00:30:06.563686 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:06.561155 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz_odh-ai-gateway-infra(9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" Nov 03 00:30:06.563686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:06.563480 5144 scope.go:122] "RemoveContainer" containerID="367acb98f496d41c2c4ecaf00c73645cd5ca635c9619a23bb97ad9b35779f928" Nov 03 00:30:06.563948 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:06.563907 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj_odh-ai-gateway-infra(fbf00a1f-da54-4267-8e1f-6b94c90b0b8a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" Nov 03 00:30:06.565899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:06.565875 5144 scope.go:122] "RemoveContainer" containerID="5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a" Nov 03 00:30:06.566142 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:06.566118 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf_odh-ai-gateway-infra(3fdd47aa-8edd-4563-b1a6-6a6f6977e893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" Nov 03 00:30:07.817363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:07.817333 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:07.987480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:07.987392 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw\" (UniqueName: \"kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw\") pod \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " Nov 03 00:30:07.987689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:07.987560 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca\") pod \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\" (UID: \"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a\") " Nov 03 00:30:07.988483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:07.987911 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca" pod "fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" (UID: "fbf00a1f-da54-4267-8e1f-6b94c90b0b8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:07.988687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:07.988605 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:07.993690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:07.993622 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw" pod "fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" (UID: "fbf00a1f-da54-4267-8e1f-6b94c90b0b8a"). InnerVolumeSpecName "kube-api-access-p7rsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:08.089889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:08.089857 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7rsw\" (UniqueName: \"kubernetes.io/projected/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a-kube-api-access-p7rsw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:08.587669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:08.587619 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" Nov 03 00:30:08.587831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:08.587680 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj" event={"ID":"fbf00a1f-da54-4267-8e1f-6b94c90b0b8a","Type":"ContainerDied","Data":"fe1f0cd9d4b2d7ce8efa85f32b27b726d01d0ecf79d0751ccd9a740909a68a3a"} Nov 03 00:30:08.587831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:08.587745 5144 scope.go:122] "RemoveContainer" containerID="367acb98f496d41c2c4ecaf00c73645cd5ca635c9619a23bb97ad9b35779f928" Nov 03 00:30:08.756891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:08.756861 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj"] Nov 03 00:30:08.805570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:08.805542 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-fd6ea3-29894430-n42fj"] Nov 03 00:30:10.008096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.008062 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" path="/var/lib/kubelet/pods/fbf00a1f-da54-4267-8e1f-6b94c90b0b8a/volumes" Nov 03 00:30:10.370471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.370399 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:30:10.371244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.371224 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:30:10.827167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.827131 5144 scope.go:122] "RemoveContainer" containerID="317bc3032035ee6dbde6f2c5f8254da90d09c73f62fa704c652a019c2b3a1d3f" Nov 03 00:30:10.837998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.837978 5144 scope.go:122] "RemoveContainer" containerID="ea4d32534dfe71dca6326e5f8cc06634200875b8dabeb497443c16af643b037a" Nov 03 00:30:10.848133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.848114 5144 scope.go:122] "RemoveContainer" containerID="1d8b07a181bee37197b6955789c0155130575fd9c5f567d4dc9f06718a4711f7" Nov 03 00:30:10.857747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.857729 5144 scope.go:122] "RemoveContainer" containerID="0e6e5aa13bb58639b0ebb43301cd6d9808735677320007bad17f06d97bb17a17" Nov 03 00:30:10.867257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.867234 5144 scope.go:122] "RemoveContainer" containerID="00ad5f52de7df46aaa87c515a71d34fa8bfa3fa8274403ec87a50f5411f20b8e" Nov 03 00:30:10.879154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.879133 5144 scope.go:122] "RemoveContainer" containerID="f9a3a5d8073634c4711f223bfbe54622fc778e8cec731373f3697c28d67ae079" Nov 03 00:30:10.889493 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:10.889476 5144 scope.go:122] "RemoveContainer" containerID="3322db2d1355865bbb36ac758f299ce7004b6007bd5c62a4cbcfe27afead9541" Nov 03 00:30:11.073757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.073727 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" podUID="2d420a25-f922-4d04-94bf-01ff94d4e71c" containerName="payload-processing" containerID="cri-o://432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06" gracePeriod=25 Nov 03 00:30:11.392959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.392933 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:30:11.544460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.544428 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume\") pod \"2d420a25-f922-4d04-94bf-01ff94d4e71c\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " Nov 03 00:30:11.544829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.544740 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w\" (UniqueName: \"kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w\") pod \"2d420a25-f922-4d04-94bf-01ff94d4e71c\" (UID: \"2d420a25-f922-4d04-94bf-01ff94d4e71c\") " Nov 03 00:30:11.544954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.544839 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume" pod "2d420a25-f922-4d04-94bf-01ff94d4e71c" (UID: "2d420a25-f922-4d04-94bf-01ff94d4e71c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:11.547350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.547276 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w" pod "2d420a25-f922-4d04-94bf-01ff94d4e71c" (UID: "2d420a25-f922-4d04-94bf-01ff94d4e71c"). InnerVolumeSpecName "kube-api-access-mjx6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:11.557068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.557042 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d420a25-f922-4d04-94bf-01ff94d4e71c-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:11.557141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.557072 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjx6w\" (UniqueName: \"kubernetes.io/projected/2d420a25-f922-4d04-94bf-01ff94d4e71c-kube-api-access-mjx6w\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:11.621398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.621325 5144 generic.go:356] "Generic (PLEG): container finished" podID="2d420a25-f922-4d04-94bf-01ff94d4e71c" containerID="432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06" exitCode=0 Nov 03 00:30:11.621398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.621369 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" Nov 03 00:30:11.621587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.621408 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" event={"ID":"2d420a25-f922-4d04-94bf-01ff94d4e71c","Type":"ContainerDied","Data":"432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06"} Nov 03 00:30:11.621587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.621457 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm" event={"ID":"2d420a25-f922-4d04-94bf-01ff94d4e71c","Type":"ContainerDied","Data":"e54818c370f1533f5b960dd1fcc100f9aa9b945d78b01664d6b535c2d5351c7f"} Nov 03 00:30:11.621587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.621486 5144 scope.go:122] "RemoveContainer" containerID="432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06" Nov 03 00:30:11.633106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.633088 5144 scope.go:122] "RemoveContainer" containerID="432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06" Nov 03 00:30:11.633404 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:11.633386 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06\": container with ID starting with 432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06 not found: ID does not exist" containerID="432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06" Nov 03 00:30:11.633445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.633412 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06"} err="failed to get container status \"432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06\": rpc error: code = NotFound desc = could not find container \"432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06\": container with ID starting with 432de760e535b984163a15af9dd0a9696f140a7b31d66a1570acde986b94bf06 not found: ID does not exist" Nov 03 00:30:11.788503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.788466 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm"] Nov 03 00:30:11.793336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.793313 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-fd6ea3-65dc85798c-4cbpm"] Nov 03 00:30:11.997870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:11.997796 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d420a25-f922-4d04-94bf-01ff94d4e71c" path="/var/lib/kubelet/pods/2d420a25-f922-4d04-94bf-01ff94d4e71c/volumes" Nov 03 00:30:12.160838 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:12.160804 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:30:14.989438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:14.989403 5144 scope.go:122] "RemoveContainer" containerID="d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613" Nov 03 00:30:15.697727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:15.697565 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerStarted","Data":"090e6d275c05f82ee364d7395c7bdb2bc029465eb40ebb26addefc8d1d389d30"} Nov 03 00:30:15.989458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:15.989395 5144 scope.go:122] "RemoveContainer" containerID="5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a" Nov 03 00:30:15.989458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:15.989441 5144 scope.go:122] "RemoveContainer" containerID="012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8" Nov 03 00:30:16.708043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.707957 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerStarted","Data":"46a3363ec83fdad5e28cdf5cbe54cc38e553529ac14d33a047b86f7a51a2cf72"} Nov 03 00:30:16.710442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.710412 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerStarted","Data":"ed6e69a996d44b82b0a560c4c6be5dfbe3f0169854c204eb60e57cae82f3a464"} Nov 03 00:30:16.713046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.713023 5144 generic.go:356] "Generic (PLEG): container finished" podID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerID="090e6d275c05f82ee364d7395c7bdb2bc029465eb40ebb26addefc8d1d389d30" exitCode=60 Nov 03 00:30:16.713124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.713096 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerDied","Data":"090e6d275c05f82ee364d7395c7bdb2bc029465eb40ebb26addefc8d1d389d30"} Nov 03 00:30:16.713173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.713128 5144 scope.go:122] "RemoveContainer" containerID="d06db0ab45588749fb5274c222fddb4185f7bcc28d8021c358fb8bf9704e8613" Nov 03 00:30:16.713437 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.713424 5144 scope.go:122] "RemoveContainer" containerID="090e6d275c05f82ee364d7395c7bdb2bc029465eb40ebb26addefc8d1d389d30" Nov 03 00:30:16.713786 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:16.713761 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs_odh-ai-gateway-infra(3a7f037a-ae59-4ef2-8580-824e5b2cecf2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" Nov 03 00:30:16.835817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.835780 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs"] Nov 03 00:30:16.989584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.989513 5144 scope.go:122] "RemoveContainer" containerID="d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a" Nov 03 00:30:16.989965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:16.989725 5144 scope.go:122] "RemoveContainer" containerID="5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a" Nov 03 00:30:17.727945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.727855 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerStarted","Data":"59f3b4f9528f8e77ce081b29a48cfcc03d34fe062a1a9ff3ab10ac545a31f8fd"} Nov 03 00:30:17.730448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.730417 5144 generic.go:356] "Generic (PLEG): container finished" podID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerID="ed6e69a996d44b82b0a560c4c6be5dfbe3f0169854c204eb60e57cae82f3a464" exitCode=60 Nov 03 00:30:17.730609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.730491 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerDied","Data":"ed6e69a996d44b82b0a560c4c6be5dfbe3f0169854c204eb60e57cae82f3a464"} Nov 03 00:30:17.730609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.730550 5144 scope.go:122] "RemoveContainer" containerID="5c6a7a7f0b87392da4bf442fa3078e0fe4d52bd52de4667809d4e89583b2b61a" Nov 03 00:30:17.730964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.730944 5144 scope.go:122] "RemoveContainer" containerID="ed6e69a996d44b82b0a560c4c6be5dfbe3f0169854c204eb60e57cae82f3a464" Nov 03 00:30:17.731522 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:17.731300 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz_odh-ai-gateway-infra(9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" Nov 03 00:30:17.737216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.736298 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerStarted","Data":"5bf64fc804ac763c950a3f733d5a0127e1a6ba6a54ea39d10a0fefca6850a10a"} Nov 03 00:30:17.739893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.739735 5144 generic.go:356] "Generic (PLEG): container finished" podID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerID="46a3363ec83fdad5e28cdf5cbe54cc38e553529ac14d33a047b86f7a51a2cf72" exitCode=60 Nov 03 00:30:17.739893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.739771 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerDied","Data":"46a3363ec83fdad5e28cdf5cbe54cc38e553529ac14d33a047b86f7a51a2cf72"} Nov 03 00:30:17.740786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.740159 5144 scope.go:122] "RemoveContainer" containerID="46a3363ec83fdad5e28cdf5cbe54cc38e553529ac14d33a047b86f7a51a2cf72" Nov 03 00:30:17.740786 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:17.740563 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz_odh-ai-gateway-infra(1241378f-7ae1-468b-98cf-2b69455d4a63)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" Nov 03 00:30:17.749354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.749333 5144 scope.go:122] "RemoveContainer" containerID="012035e3d9e7f35c012519e1983b64221b1381734ea6d6b9e902f517cb2bb0a8" Nov 03 00:30:17.888165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.886438 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz"] Nov 03 00:30:17.955125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:17.955100 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:18.065601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.065576 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz"] Nov 03 00:30:18.069424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.069401 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx\" (UniqueName: \"kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx\") pod \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " Nov 03 00:30:18.069544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.069528 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca\") pod \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\" (UID: \"3a7f037a-ae59-4ef2-8580-824e5b2cecf2\") " Nov 03 00:30:18.069948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.069921 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca" pod "3a7f037a-ae59-4ef2-8580-824e5b2cecf2" (UID: "3a7f037a-ae59-4ef2-8580-824e5b2cecf2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:18.070596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.070574 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:18.071617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.071601 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx" pod "3a7f037a-ae59-4ef2-8580-824e5b2cecf2" (UID: "3a7f037a-ae59-4ef2-8580-824e5b2cecf2"). InnerVolumeSpecName "kube-api-access-q6xdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:18.171716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.171691 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6xdx\" (UniqueName: \"kubernetes.io/projected/3a7f037a-ae59-4ef2-8580-824e5b2cecf2-kube-api-access-q6xdx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:18.752684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.752457 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" Nov 03 00:30:18.752684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.752462 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs" event={"ID":"3a7f037a-ae59-4ef2-8580-824e5b2cecf2","Type":"ContainerDied","Data":"bcbeb5b3e7e054a420cbf1340d31fd131b8d529b437a95fcd3bd8922b890e52b"} Nov 03 00:30:18.752684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.752565 5144 scope.go:122] "RemoveContainer" containerID="090e6d275c05f82ee364d7395c7bdb2bc029465eb40ebb26addefc8d1d389d30" Nov 03 00:30:18.755293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.755241 5144 generic.go:356] "Generic (PLEG): container finished" podID="138fa9ce-6c3b-4115-9104-993411563d70" containerID="5bf64fc804ac763c950a3f733d5a0127e1a6ba6a54ea39d10a0fefca6850a10a" exitCode=60 Nov 03 00:30:18.755482 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.755458 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerDied","Data":"5bf64fc804ac763c950a3f733d5a0127e1a6ba6a54ea39d10a0fefca6850a10a"} Nov 03 00:30:18.756266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.756246 5144 scope.go:122] "RemoveContainer" containerID="5bf64fc804ac763c950a3f733d5a0127e1a6ba6a54ea39d10a0fefca6850a10a" Nov 03 00:30:18.757005 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:18.756589 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht_odh-ai-gateway-infra(138fa9ce-6c3b-4115-9104-993411563d70)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" podUID="138fa9ce-6c3b-4115-9104-993411563d70" Nov 03 00:30:18.776433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.776406 5144 scope.go:122] "RemoveContainer" containerID="d494ffffb9fd73e4d9f4b60bbd1d874eaeb193bee0d1909161f43f364aaf422a" Nov 03 00:30:18.776612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.776571 5144 generic.go:356] "Generic (PLEG): container finished" podID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerID="59f3b4f9528f8e77ce081b29a48cfcc03d34fe062a1a9ff3ab10ac545a31f8fd" exitCode=60 Nov 03 00:30:18.776754 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.776731 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerDied","Data":"59f3b4f9528f8e77ce081b29a48cfcc03d34fe062a1a9ff3ab10ac545a31f8fd"} Nov 03 00:30:18.777243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.777225 5144 scope.go:122] "RemoveContainer" containerID="59f3b4f9528f8e77ce081b29a48cfcc03d34fe062a1a9ff3ab10ac545a31f8fd" Nov 03 00:30:18.777618 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:18.777581 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf_odh-ai-gateway-infra(3fdd47aa-8edd-4563-b1a6-6a6f6977e893)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" Nov 03 00:30:18.797981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.797957 5144 scope.go:122] "RemoveContainer" containerID="5f56495546cd7c99fdf3e3d9bb828e3830b46cadb367463b9d71fa4da55de22a" Nov 03 00:30:18.856214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.856156 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf"] Nov 03 00:30:18.956195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.956171 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:18.965260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.965239 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:18.989082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:18.989063 5144 scope.go:122] "RemoveContainer" containerID="a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509" Nov 03 00:30:19.092909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.092863 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp\" (UniqueName: \"kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp\") pod \"1241378f-7ae1-468b-98cf-2b69455d4a63\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " Nov 03 00:30:19.093333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.092930 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca\") pod \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " Nov 03 00:30:19.093333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.093114 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca\") pod \"1241378f-7ae1-468b-98cf-2b69455d4a63\" (UID: \"1241378f-7ae1-468b-98cf-2b69455d4a63\") " Nov 03 00:30:19.093333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.093151 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd\" (UniqueName: \"kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd\") pod \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\" (UID: \"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80\") " Nov 03 00:30:19.093885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.093860 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca" pod "9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" (UID: "9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:19.094517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.094495 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs"] Nov 03 00:30:19.094885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.094853 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca" pod "1241378f-7ae1-468b-98cf-2b69455d4a63" (UID: "1241378f-7ae1-468b-98cf-2b69455d4a63"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:19.096296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.096268 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp" pod "1241378f-7ae1-468b-98cf-2b69455d4a63" (UID: "1241378f-7ae1-468b-98cf-2b69455d4a63"). InnerVolumeSpecName "kube-api-access-4pnnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:19.097201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.097168 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd" pod "9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" (UID: "9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80"). InnerVolumeSpecName "kube-api-access-fqsqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:19.099309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.099288 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-be578d-29894430-grbhs"] Nov 03 00:30:19.194937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.194908 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1241378f-7ae1-468b-98cf-2b69455d4a63-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.194937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.194934 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqsqd\" (UniqueName: \"kubernetes.io/projected/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-kube-api-access-fqsqd\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.195133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.194945 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pnnp\" (UniqueName: \"kubernetes.io/projected/1241378f-7ae1-468b-98cf-2b69455d4a63-kube-api-access-4pnnp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.195133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.194954 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:19.206950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.206925 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht"] Nov 03 00:30:19.787975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.787941 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerStarted","Data":"0baa576c27d1ee1f6157801871033f5994b263c7865dc8384aa7ba372db7a81c"} Nov 03 00:30:19.789492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.789470 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" Nov 03 00:30:19.789492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.789478 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz" event={"ID":"9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80","Type":"ContainerDied","Data":"c70170d4224d5155d443dc5e673b161eceb3f920e21733e8bed7347254736be0"} Nov 03 00:30:19.789738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.789602 5144 scope.go:122] "RemoveContainer" containerID="ed6e69a996d44b82b0a560c4c6be5dfbe3f0169854c204eb60e57cae82f3a464" Nov 03 00:30:19.794460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.794443 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" Nov 03 00:30:19.794558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.794482 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz" event={"ID":"1241378f-7ae1-468b-98cf-2b69455d4a63","Type":"ContainerDied","Data":"74ca0f319ab8374745784016a5e5f7dbefd828a6c1260a83985b871be502828d"} Nov 03 00:30:19.802947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:19.802762 5144 scope.go:122] "RemoveContainer" containerID="46a3363ec83fdad5e28cdf5cbe54cc38e553529ac14d33a047b86f7a51a2cf72" Nov 03 00:30:20.000353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.000323 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" path="/var/lib/kubelet/pods/3a7f037a-ae59-4ef2-8580-824e5b2cecf2/volumes" Nov 03 00:30:20.021142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.021117 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:20.030465 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.030438 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz"] Nov 03 00:30:20.036099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.036077 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894430-bp6gz"] Nov 03 00:30:20.088756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.088732 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:20.116304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.116273 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca\") pod \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " Nov 03 00:30:20.116620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.116554 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca" pod "3fdd47aa-8edd-4563-b1a6-6a6f6977e893" (UID: "3fdd47aa-8edd-4563-b1a6-6a6f6977e893"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:20.116926 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.116905 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6\" (UniqueName: \"kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6\") pod \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\" (UID: \"3fdd47aa-8edd-4563-b1a6-6a6f6977e893\") " Nov 03 00:30:20.117476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.117456 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:20.119277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.119250 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6" pod "3fdd47aa-8edd-4563-b1a6-6a6f6977e893" (UID: "3fdd47aa-8edd-4563-b1a6-6a6f6977e893"). InnerVolumeSpecName "kube-api-access-cv9n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:20.218881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.218851 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca\") pod \"138fa9ce-6c3b-4115-9104-993411563d70\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " Nov 03 00:30:20.219050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.218901 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79\" (UniqueName: \"kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79\") pod \"138fa9ce-6c3b-4115-9104-993411563d70\" (UID: \"138fa9ce-6c3b-4115-9104-993411563d70\") " Nov 03 00:30:20.219178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.219156 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca" pod "138fa9ce-6c3b-4115-9104-993411563d70" (UID: "138fa9ce-6c3b-4115-9104-993411563d70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:20.219432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.219413 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv9n6\" (UniqueName: \"kubernetes.io/projected/3fdd47aa-8edd-4563-b1a6-6a6f6977e893-kube-api-access-cv9n6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:20.219432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.219433 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/138fa9ce-6c3b-4115-9104-993411563d70-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:20.221203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.221173 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79" pod "138fa9ce-6c3b-4115-9104-993411563d70" (UID: "138fa9ce-6c3b-4115-9104-993411563d70"). InnerVolumeSpecName "kube-api-access-tfs79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:20.307457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.307428 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w"] Nov 03 00:30:20.307702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.307674 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" containerID="cri-o://5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef" gracePeriod=30 Nov 03 00:30:20.321198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.321169 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tfs79\" (UniqueName: \"kubernetes.io/projected/138fa9ce-6c3b-4115-9104-993411563d70-kube-api-access-tfs79\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:20.363521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.363465 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz"] Nov 03 00:30:20.367502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.367478 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-cb65f3a5-29894430-zhxzz"] Nov 03 00:30:20.807690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.807666 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" Nov 03 00:30:20.807855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.807694 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht" event={"ID":"138fa9ce-6c3b-4115-9104-993411563d70","Type":"ContainerDied","Data":"4928cf343d254d0268db1b979af663b4aee67951cb279a57391757a98b58f563"} Nov 03 00:30:20.807855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.807752 5144 scope.go:122] "RemoveContainer" containerID="5bf64fc804ac763c950a3f733d5a0127e1a6ba6a54ea39d10a0fefca6850a10a" Nov 03 00:30:20.810688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.810660 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" event={"ID":"3fdd47aa-8edd-4563-b1a6-6a6f6977e893","Type":"ContainerDied","Data":"83dc158631b83d0c022fa2755635ca47b83b5928c597c765b66f5e0c78f211ef"} Nov 03 00:30:20.810688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.810683 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf" Nov 03 00:30:20.818380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.818171 5144 generic.go:356] "Generic (PLEG): container finished" podID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerID="0baa576c27d1ee1f6157801871033f5994b263c7865dc8384aa7ba372db7a81c" exitCode=60 Nov 03 00:30:20.818380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.818235 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerDied","Data":"0baa576c27d1ee1f6157801871033f5994b263c7865dc8384aa7ba372db7a81c"} Nov 03 00:30:20.819157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.818667 5144 scope.go:122] "RemoveContainer" containerID="0baa576c27d1ee1f6157801871033f5994b263c7865dc8384aa7ba372db7a81c" Nov 03 00:30:20.819157 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:20.818914 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp_odh-ai-gateway-infra(1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" Nov 03 00:30:20.819934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.819920 5144 scope.go:122] "RemoveContainer" containerID="59f3b4f9528f8e77ce081b29a48cfcc03d34fe062a1a9ff3ab10ac545a31f8fd" Nov 03 00:30:20.829499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.829482 5144 scope.go:122] "RemoveContainer" containerID="a392b7eb09d7c1fe8d49bf07c4ae4bba95ad484aed9b4369e38345922fa8f509" Nov 03 00:30:20.917007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.916982 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp"] Nov 03 00:30:20.976559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.976530 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf"] Nov 03 00:30:20.982235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:20.982213 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894430-5zdkf"] Nov 03 00:30:21.093548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:21.093480 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht"] Nov 03 00:30:21.097658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:21.097624 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894430-gtnht"] Nov 03 00:30:22.041111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.041084 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" path="/var/lib/kubelet/pods/1241378f-7ae1-468b-98cf-2b69455d4a63/volumes" Nov 03 00:30:22.042044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.042024 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="138fa9ce-6c3b-4115-9104-993411563d70" path="/var/lib/kubelet/pods/138fa9ce-6c3b-4115-9104-993411563d70/volumes" Nov 03 00:30:22.042550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.042535 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" path="/var/lib/kubelet/pods/3fdd47aa-8edd-4563-b1a6-6a6f6977e893/volumes" Nov 03 00:30:22.043372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.043357 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" path="/var/lib/kubelet/pods/9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80/volumes" Nov 03 00:30:22.251850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.251819 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:22.444250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.444176 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq\" (UniqueName: \"kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq\") pod \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " Nov 03 00:30:22.444409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.444389 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca\") pod \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\" (UID: \"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6\") " Nov 03 00:30:22.444757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.444733 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca" pod "1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" (UID: "1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:22.445029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.445016 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:22.446418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.446401 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq" pod "1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" (UID: "1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6"). InnerVolumeSpecName "kube-api-access-zrkbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:22.546575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:22.546547 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrkbq\" (UniqueName: \"kubernetes.io/projected/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6-kube-api-access-zrkbq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:23.054680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.054058 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" Nov 03 00:30:23.054680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.054367 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp" event={"ID":"1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6","Type":"ContainerDied","Data":"c27c045b4ab7d06b50563644d0d73d420258229b6d685bf49f86000ceff78117"} Nov 03 00:30:23.054680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.054476 5144 scope.go:122] "RemoveContainer" containerID="0baa576c27d1ee1f6157801871033f5994b263c7865dc8384aa7ba372db7a81c" Nov 03 00:30:23.242146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.242107 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp"] Nov 03 00:30:23.274988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.274959 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894430-hnrhp"] Nov 03 00:30:23.705793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.705764 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2"] Nov 03 00:30:23.705980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.705949 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" containerID="cri-o://dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5" gracePeriod=30 Nov 03 00:30:23.997729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:23.997655 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" path="/var/lib/kubelet/pods/1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6/volumes" Nov 03 00:30:24.045514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:24.045485 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2"] Nov 03 00:30:24.049357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:24.049335 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-000e56f7-z44x2"] Nov 03 00:30:24.233729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:24.233694 5144 patch_prober.go:29] interesting pod/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:30:24.234163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:24.233756 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:30:25.584769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.584714 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:30:25.597027 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597005 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597049 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597088 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597179 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597216 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597300 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597355 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597410 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597461 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:25.597986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597538 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs\") pod \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\" (UID: \"d5a546bd-5baf-470c-9f9a-bddbf2ecead9\") " Nov 03 00:30:25.597986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597592 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:25.597986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597750 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:25.597986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597892 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:25.598410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.597982 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:25.598410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.598311 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.598410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.598327 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.598410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.598348 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.598410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.598356 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.598410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.598365 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.599796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.599772 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "kube-api-access-7rzfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:25.599898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.599858 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:25.599941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.599911 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:25.599941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.599924 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy" pod "d5a546bd-5baf-470c-9f9a-bddbf2ecead9" (UID: "d5a546bd-5baf-470c-9f9a-bddbf2ecead9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:25.699321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.699295 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.699321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.699318 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.699462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.699328 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.699462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.699337 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rzfp\" (UniqueName: \"kubernetes.io/projected/d5a546bd-5baf-470c-9f9a-bddbf2ecead9-kube-api-access-7rzfp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:25.999290 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:25.999203 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3" path="/var/lib/kubelet/pods/f9b184a0-3764-4cb3-bdde-71cb8e9d5fc3/volumes" Nov 03 00:30:26.051539 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.051505 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6"] Nov 03 00:30:26.060168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.060132 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f532143a-pt5t6"] Nov 03 00:30:26.082135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.082107 5144 generic.go:356] "Generic (PLEG): container finished" podID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerID="5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef" exitCode=0 Nov 03 00:30:26.082294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.082155 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" Nov 03 00:30:26.082294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.082181 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" event={"ID":"d5a546bd-5baf-470c-9f9a-bddbf2ecead9","Type":"ContainerDied","Data":"5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef"} Nov 03 00:30:26.082294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.082269 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w" event={"ID":"d5a546bd-5baf-470c-9f9a-bddbf2ecead9","Type":"ContainerDied","Data":"0610860be27a2dfc2df9ca3b4ad120aa1e56979eb237c129bf8b7d8893e0c2e3"} Nov 03 00:30:26.082505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.082298 5144 scope.go:122] "RemoveContainer" containerID="5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef" Nov 03 00:30:26.093978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.093958 5144 scope.go:122] "RemoveContainer" containerID="ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0" Nov 03 00:30:26.104345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.104326 5144 scope.go:122] "RemoveContainer" containerID="5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef" Nov 03 00:30:26.104668 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:26.104639 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef\": container with ID starting with 5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef not found: ID does not exist" containerID="5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef" Nov 03 00:30:26.104751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.104666 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef"} err="failed to get container status \"5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef\": rpc error: code = NotFound desc = could not find container \"5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef\": container with ID starting with 5282f3ccd6b85144565e6b8910bee4bc2a8286f9fade541dafff6441b464efef not found: ID does not exist" Nov 03 00:30:26.104751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.104685 5144 scope.go:122] "RemoveContainer" containerID="ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0" Nov 03 00:30:26.104956 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:26.104936 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0\": container with ID starting with ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0 not found: ID does not exist" containerID="ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0" Nov 03 00:30:26.105016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.104966 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0"} err="failed to get container status \"ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0\": rpc error: code = NotFound desc = could not find container \"ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0\": container with ID starting with ece6f55a7a72eebc2daed538af417d32820b822f162270dca07f614e0ce75ae0 not found: ID does not exist" Nov 03 00:30:26.251826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.251752 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w"] Nov 03 00:30:26.259690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.259654 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-fd6ea3-openshift-default-7f4644c64f-48p2w"] Nov 03 00:30:26.325161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.325133 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm"] Nov 03 00:30:26.325381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.325358 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" containerID="cri-o://eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330" gracePeriod=30 Nov 03 00:30:26.330976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.330949 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh"] Nov 03 00:30:26.331182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:26.331160 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" containerID="cri-o://ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e" gracePeriod=30 Nov 03 00:30:27.996975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:27.996942 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60" path="/var/lib/kubelet/pods/a330dd9f-7cb9-4e46-83cb-1dc9f16a1d60/volumes" Nov 03 00:30:27.997439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:27.997425 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" path="/var/lib/kubelet/pods/d5a546bd-5baf-470c-9f9a-bddbf2ecead9/volumes" Nov 03 00:30:29.013407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.013381 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043481 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043527 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043676 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043712 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043764 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043828 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043903 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.043975 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.045143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.044023 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy\") pod \"36e621a1-9466-4ae0-a014-38621b0175de\" (UID: \"36e621a1-9466-4ae0-a014-38621b0175de\") " Nov 03 00:30:29.046434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.046282 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:29.046830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.046544 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.048002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.047383 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.048002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.047734 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.048251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.048220 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.049025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.049002 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:29.049679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.049285 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:29.049679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.049320 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "kube-api-access-z5prh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:29.049679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.049584 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy" pod "36e621a1-9466-4ae0-a014-38621b0175de" (UID: "36e621a1-9466-4ae0-a014-38621b0175de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:29.114394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.114365 5144 generic.go:356] "Generic (PLEG): container finished" podID="36e621a1-9466-4ae0-a014-38621b0175de" containerID="dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5" exitCode=0 Nov 03 00:30:29.114545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.114420 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" Nov 03 00:30:29.114545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.114471 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" event={"ID":"36e621a1-9466-4ae0-a014-38621b0175de","Type":"ContainerDied","Data":"dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5"} Nov 03 00:30:29.114545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.114530 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2" event={"ID":"36e621a1-9466-4ae0-a014-38621b0175de","Type":"ContainerDied","Data":"b23d05c8b0f715e82b135414179a3d385b2ae65e7dfa945b2ff820f23b30c863"} Nov 03 00:30:29.114760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.114552 5144 scope.go:122] "RemoveContainer" containerID="dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5" Nov 03 00:30:29.127045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.127028 5144 scope.go:122] "RemoveContainer" containerID="dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5" Nov 03 00:30:29.127382 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:29.127357 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5\": container with ID starting with dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5 not found: ID does not exist" containerID="dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5" Nov 03 00:30:29.127436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.127391 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5"} err="failed to get container status \"dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5\": rpc error: code = NotFound desc = could not find container \"dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5\": container with ID starting with dddb45b42072021aa30cde8ea00bc1b5aaab908bbbd21a25f7349ae2194031b5 not found: ID does not exist" Nov 03 00:30:29.146378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146347 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146372 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146389 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146402 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/36e621a1-9466-4ae0-a014-38621b0175de-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146413 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146426 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146439 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/36e621a1-9466-4ae0-a014-38621b0175de-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146452 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5prh\" (UniqueName: \"kubernetes.io/projected/36e621a1-9466-4ae0-a014-38621b0175de-kube-api-access-z5prh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.146526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.146465 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/36e621a1-9466-4ae0-a014-38621b0175de-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:29.292586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.292555 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2"] Nov 03 00:30:29.301568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.301525 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-xjlx2"] Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.693677 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn"] Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694767 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694792 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694808 5144 cpu_manager.go:414] "containerMap: removing container" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694818 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694836 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694844 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694865 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2d420a25-f922-4d04-94bf-01ff94d4e71c" containerName="payload-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694871 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d420a25-f922-4d04-94bf-01ff94d4e71c" containerName="payload-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694887 5144 cpu_manager.go:414] "containerMap: removing container" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694895 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694914 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694923 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694934 5144 cpu_manager.go:414] "containerMap: removing container" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694943 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694959 5144 cpu_manager.go:414] "containerMap: removing container" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694964 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694984 5144 cpu_manager.go:414] "containerMap: removing container" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694990 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.694999 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695005 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695016 5144 cpu_manager.go:414] "containerMap: removing container" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695022 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695029 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695035 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695043 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695049 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695066 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695073 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695090 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695100 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695119 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695125 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695137 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695144 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695154 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695160 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695170 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695176 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695199 5144 cpu_manager.go:414] "containerMap: removing container" podUID="de9530be-8f13-47e4-a5b9-d18c645d243f" containerName="payload-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695205 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de9530be-8f13-47e4-a5b9-d18c645d243f" containerName="payload-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695218 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695224 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695233 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a7235f6d-4796-4b2c-af54-0f13aab7bbea" containerName="payload-pre-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695241 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7235f6d-4796-4b2c-af54-0f13aab7bbea" containerName="payload-pre-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695422 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695435 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695445 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695453 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695464 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="de9530be-8f13-47e4-a5b9-d18c645d243f" containerName="payload-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695472 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695481 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7235f6d-4796-4b2c-af54-0f13aab7bbea" containerName="payload-pre-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695491 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695501 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695509 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695516 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695525 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695537 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d420a25-f922-4d04-94bf-01ff94d4e71c" containerName="payload-processing" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695546 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695555 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695562 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf00a1f-da54-4267-8e1f-6b94c90b0b8a" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695572 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="36e621a1-9466-4ae0-a014-38621b0175de" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695582 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695589 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695597 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="138fa9ce-6c3b-4115-9104-993411563d70" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695605 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1241378f-7ae1-468b-98cf-2b69455d4a63" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695616 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.699748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.695646 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.708938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.708907 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.763282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.763244 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.763758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.763733 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2qqf\" (UniqueName: \"kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf\") pod \"payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.764890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.764869 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bb5643\"" Nov 03 00:30:29.765072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.765054 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bb5643-dockercfg-fdsms\"" Nov 03 00:30:29.776943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.776919 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn"] Nov 03 00:30:29.820510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.819682 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs"] Nov 03 00:30:29.820746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820715 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" Nov 03 00:30:29.820746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820735 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a546bd-5baf-470c-9f9a-bddbf2ecead9" containerName="istio-proxy" Nov 03 00:30:29.820874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820789 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.820874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820797 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a7f037a-ae59-4ef2-8580-824e5b2cecf2" containerName="cleanup" Nov 03 00:30:29.820874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820814 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.820874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820822 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.821089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820961 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.821089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.820970 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.821186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.821140 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fdd47aa-8edd-4563-b1a6-6a6f6977e893" containerName="cleanup" Nov 03 00:30:29.821186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.821156 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffd5a64-ffbd-4c8a-9d85-7626c6c226a6" containerName="cleanup" Nov 03 00:30:29.821186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.821168 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3357bb-1ff9-4e8f-bc9b-4c23f48a7a80" containerName="cleanup" Nov 03 00:30:29.825622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.825602 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:29.865059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.865033 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbr7\" (UniqueName: \"kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:29.865237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.865103 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2qqf\" (UniqueName: \"kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf\") pod \"payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.865237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.865202 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:29.865353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.865334 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.865966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.865948 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.887734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.887706 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs"] Nov 03 00:30:29.921373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.921334 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2qqf\" (UniqueName: \"kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf\") pod \"payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:29.966299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.966232 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbr7\" (UniqueName: \"kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:29.966423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.966375 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:29.967029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.967007 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:29.997942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:29.997918 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36e621a1-9466-4ae0-a014-38621b0175de" path="/var/lib/kubelet/pods/36e621a1-9466-4ae0-a014-38621b0175de/volumes" Nov 03 00:30:30.022936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:30.022905 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbr7\" (UniqueName: \"kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7\") pod \"payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:30.065055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:30.065030 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:30.143933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:30.143115 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:30.446898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:30.446867 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn"] Nov 03 00:30:30.447088 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:30.447068 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678b16e7_6c48_4246_81ae_134a663e6046.slice/crio-9bc29372a39ddc675a25e9b7809be6a7885ae3b3f7a55258c436c027d8433cd9 WatchSource:0}: Error finding container 9bc29372a39ddc675a25e9b7809be6a7885ae3b3f7a55258c436c027d8433cd9: Status 404 returned error can't find the container with id 9bc29372a39ddc675a25e9b7809be6a7885ae3b3f7a55258c436c027d8433cd9 Nov 03 00:30:30.561267 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:30.561235 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbd23a01_4668_4afb_8116_f35c52087662.slice/crio-a3347f8eb3fdb63412b3b5ca5885cbb49794100d5281f59dfa3113bdc45eb762 WatchSource:0}: Error finding container a3347f8eb3fdb63412b3b5ca5885cbb49794100d5281f59dfa3113bdc45eb762: Status 404 returned error can't find the container with id a3347f8eb3fdb63412b3b5ca5885cbb49794100d5281f59dfa3113bdc45eb762 Nov 03 00:30:30.561904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:30.561881 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs"] Nov 03 00:30:31.046583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.046549 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5"] Nov 03 00:30:31.053662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.053621 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-21ae6c16-h5hq5"] Nov 03 00:30:31.161087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.161038 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" event={"ID":"cbd23a01-4668-4afb-8116-f35c52087662","Type":"ContainerStarted","Data":"f40d1b1934b41ac3240c318df3ca9667e415e9b3d56cc7baa954b29a13fca062"} Nov 03 00:30:31.161261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.161117 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" event={"ID":"cbd23a01-4668-4afb-8116-f35c52087662","Type":"ContainerStarted","Data":"a3347f8eb3fdb63412b3b5ca5885cbb49794100d5281f59dfa3113bdc45eb762"} Nov 03 00:30:31.162962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.162935 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" event={"ID":"678b16e7-6c48-4246-81ae-134a663e6046","Type":"ContainerStarted","Data":"3fdce939e8fd90b258b76c9fb06c3c0c192e0e7192d8511beb61ef7ee28bbaf8"} Nov 03 00:30:31.163095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.162978 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" event={"ID":"678b16e7-6c48-4246-81ae-134a663e6046","Type":"ContainerStarted","Data":"9bc29372a39ddc675a25e9b7809be6a7885ae3b3f7a55258c436c027d8433cd9"} Nov 03 00:30:31.163095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.163023 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:31.275644 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.275582 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" podStartSLOduration=2.275567252 podStartE2EDuration="2.275567252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:31.272426168 +0000 UTC m=+1521.418120620" watchObservedRunningTime="2026-11-03 00:30:31.275567252 +0000 UTC m=+1521.421261709" Nov 03 00:30:31.389184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.389132 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" podStartSLOduration=2.389113051 podStartE2EDuration="2.389113051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:31.385404646 +0000 UTC m=+1521.531099124" watchObservedRunningTime="2026-11-03 00:30:31.389113051 +0000 UTC m=+1521.534807507" Nov 03 00:30:31.698375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.698351 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:30:31.701912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.701889 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:30:31.794824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.794798 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.794847 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.794892 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.794949 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.794966 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.794993 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795031 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795177 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.795288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795185 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.795288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795229 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795317 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795353 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795355 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.795497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795419 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795458 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795480 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795499 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795573 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795601 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh\") pod \"e131da3f-da7b-4026-99a9-e56a2789fb7a\" (UID: \"e131da3f-da7b-4026-99a9-e56a2789fb7a\") " Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795761 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795771 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795770 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.795845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795824 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.796192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795877 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4\") pod \"91c4ffff-f066-4253-886b-9de80300737e\" (UID: \"91c4ffff-f066-4253-886b-9de80300737e\") " Nov 03 00:30:31.796192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.795896 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:31.796192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796119 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.796192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796142 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.796611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796576 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:31.796861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796802 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796866 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e131da3f-da7b-4026-99a9-e56a2789fb7a-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796881 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796893 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796911 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796924 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796942 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.796962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796955 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.797488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.796969 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.798200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.798173 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:31.798320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.798223 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:31.798596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.798571 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "kube-api-access-dvqxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:31.798715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.798697 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:31.798757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.798724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "kube-api-access-qkmd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:31.798985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.798963 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.799170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.799153 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy" pod "e131da3f-da7b-4026-99a9-e56a2789fb7a" (UID: "e131da3f-da7b-4026-99a9-e56a2789fb7a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:31.799732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.799711 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo" pod "91c4ffff-f066-4253-886b-9de80300737e" (UID: "91c4ffff-f066-4253-886b-9de80300737e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:31.897874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897794 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91c4ffff-f066-4253-886b-9de80300737e-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.897874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897830 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkmd4\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-kube-api-access-qkmd4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.897874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897847 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91c4ffff-f066-4253-886b-9de80300737e-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.897874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897863 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.898104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897878 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.898104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897893 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e131da3f-da7b-4026-99a9-e56a2789fb7a-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.898104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897907 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91c4ffff-f066-4253-886b-9de80300737e-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.898104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897919 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91c4ffff-f066-4253-886b-9de80300737e-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.898104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.897933 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvqxh\" (UniqueName: \"kubernetes.io/projected/e131da3f-da7b-4026-99a9-e56a2789fb7a-kube-api-access-dvqxh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:31.999856 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:31.999823 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f4dfb68-7089-48f5-9253-5a833be259e2" path="/var/lib/kubelet/pods/3f4dfb68-7089-48f5-9253-5a833be259e2/volumes" Nov 03 00:30:32.172719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.172603 5144 generic.go:356] "Generic (PLEG): container finished" podID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerID="eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330" exitCode=0 Nov 03 00:30:32.172719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.172687 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" event={"ID":"e131da3f-da7b-4026-99a9-e56a2789fb7a","Type":"ContainerDied","Data":"eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330"} Nov 03 00:30:32.173228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.172697 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" Nov 03 00:30:32.173228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.172755 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm" event={"ID":"e131da3f-da7b-4026-99a9-e56a2789fb7a","Type":"ContainerDied","Data":"67c6cd3a0dae3fe7b40040e1244b7caee15b62a83e3ba9b9ef197bb8c1408478"} Nov 03 00:30:32.173228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.172796 5144 scope.go:122] "RemoveContainer" containerID="eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330" Nov 03 00:30:32.174747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.174727 5144 generic.go:356] "Generic (PLEG): container finished" podID="91c4ffff-f066-4253-886b-9de80300737e" containerID="ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e" exitCode=0 Nov 03 00:30:32.174879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.174760 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" event={"ID":"91c4ffff-f066-4253-886b-9de80300737e","Type":"ContainerDied","Data":"ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e"} Nov 03 00:30:32.174879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.174788 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" Nov 03 00:30:32.174879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.174809 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh" event={"ID":"91c4ffff-f066-4253-886b-9de80300737e","Type":"ContainerDied","Data":"f74b1fe184c479c92efd0b4cd31dbce24a1e9765662d3b88c1f7906d9084f01a"} Nov 03 00:30:32.175376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.175091 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:32.184842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.184826 5144 scope.go:122] "RemoveContainer" containerID="eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330" Nov 03 00:30:32.185114 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:32.185094 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330\": container with ID starting with eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330 not found: ID does not exist" containerID="eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330" Nov 03 00:30:32.185161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.185123 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330"} err="failed to get container status \"eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330\": rpc error: code = NotFound desc = could not find container \"eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330\": container with ID starting with eb456734e578502a0846442392034bf7ed593e7d97fb356e480ea76b8bbe7330 not found: ID does not exist" Nov 03 00:30:32.185161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.185139 5144 scope.go:122] "RemoveContainer" containerID="ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e" Nov 03 00:30:32.195086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.195071 5144 scope.go:122] "RemoveContainer" containerID="ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e" Nov 03 00:30:32.195423 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:32.195407 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e\": container with ID starting with ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e not found: ID does not exist" containerID="ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e" Nov 03 00:30:32.195474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.195429 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e"} err="failed to get container status \"ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e\": rpc error: code = NotFound desc = could not find container \"ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e\": container with ID starting with ffcefb181eb898a1b7d35fb4d3f1a673ea928af517153d961cc8abd296f70a4e not found: ID does not exist" Nov 03 00:30:32.340422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.340383 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh"] Nov 03 00:30:32.387058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.387027 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-lbnjh"] Nov 03 00:30:32.450371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.450280 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm"] Nov 03 00:30:32.456346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:32.456320 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-f9ghm"] Nov 03 00:30:33.997192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:33.997160 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c4ffff-f066-4253-886b-9de80300737e" path="/var/lib/kubelet/pods/91c4ffff-f066-4253-886b-9de80300737e/volumes" Nov 03 00:30:33.997688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:33.997673 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" path="/var/lib/kubelet/pods/e131da3f-da7b-4026-99a9-e56a2789fb7a/volumes" Nov 03 00:30:40.047612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:40.047579 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b"] Nov 03 00:30:40.097218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:40.097176 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65481551-m6r8b"] Nov 03 00:30:42.005710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:42.005672 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dff6e014-d317-47f5-a224-bc02ece6a9c4" path="/var/lib/kubelet/pods/dff6e014-d317-47f5-a224-bc02ece6a9c4/volumes" Nov 03 00:30:42.181259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:42.178426 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:30:42.189359 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:42.189328 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:30:43.190224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:43.189913 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:30:45.381477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:45.381442 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm"] Nov 03 00:30:45.428788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:45.428755 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-b94qm"] Nov 03 00:30:45.999172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:45.999135 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae77c19e-dd79-47cb-8853-5978a9650419" path="/var/lib/kubelet/pods/ae77c19e-dd79-47cb-8853-5978a9650419/volumes" Nov 03 00:30:48.339698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:48.339611 5144 generic.go:356] "Generic (PLEG): container finished" podID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerID="ae33c85f3e629291fdedc712152ee66db826d970f60cfab53a48746b159dc915" exitCode=0 Nov 03 00:30:48.340219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:48.339700 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" event={"ID":"e6f913c4-cfcf-4f8a-96ef-66f0debc631f","Type":"ContainerDied","Data":"ae33c85f3e629291fdedc712152ee66db826d970f60cfab53a48746b159dc915"} Nov 03 00:30:48.340219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:48.340159 5144 scope.go:122] "RemoveContainer" containerID="ae33c85f3e629291fdedc712152ee66db826d970f60cfab53a48746b159dc915" Nov 03 00:30:48.341757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:48.341728 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:48.341898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:48.341836 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:48.341898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:48.341882 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:30:49.350266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:49.350234 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" event={"ID":"e6f913c4-cfcf-4f8a-96ef-66f0debc631f","Type":"ContainerStarted","Data":"5083af0e35495acaedb304e31142812baa9086ba7a62418c2eef54ce64bc48a0"} Nov 03 00:30:49.484187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:49.484158 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:30:49.485041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:49.485017 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 00:30:49.485183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:49.485061 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 00:30:49.974623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:49.974591 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:30:50.484337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:50.484299 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 00:30:50.484907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:50.484363 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 00:30:51.484297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:51.484267 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 00:30:51.484467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:51.484317 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 00:30:52.484874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:52.484837 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 00:30:52.485344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:52.484898 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 00:30:53.484880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:53.484841 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" start-of-body= Nov 03 00:30:53.485361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:53.484906 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.150:15021/healthz/ready\": dial tcp 10.132.0.150:15021: connect: connection refused" Nov 03 00:30:53.553279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:53.553245 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq"] Nov 03 00:30:53.553837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:53.553809 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" containerID="cri-o://26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51" gracePeriod=30 Nov 03 00:30:54.489007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:54.488978 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:30:54.490206 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:54.490186 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:30:55.634575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:55.634538 5144 patch_prober.go:29] interesting pod/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:30:55.635079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:55.634600 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:30:56.313598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.313561 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw"] Nov 03 00:30:56.314390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.314350 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" Nov 03 00:30:56.314390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.314389 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" Nov 03 00:30:56.314568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.314457 5144 cpu_manager.go:414] "containerMap: removing container" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" Nov 03 00:30:56.314568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.314465 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" Nov 03 00:30:56.314696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.314675 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c4ffff-f066-4253-886b-9de80300737e" containerName="istio-proxy" Nov 03 00:30:56.314750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.314700 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e131da3f-da7b-4026-99a9-e56a2789fb7a" containerName="istio-proxy" Nov 03 00:30:56.317670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.317652 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.376825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.376797 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw"] Nov 03 00:30:56.419858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.419816 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-cb65f3a5\"" Nov 03 00:30:56.482682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.478457 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.482682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.478546 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qpfvs\" (UniqueName: \"kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.521711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.503586 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g"] Nov 03 00:30:56.555943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.555912 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.583955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.582395 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.584340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.584315 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qpfvs\" (UniqueName: \"kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.587976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.587932 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.597063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.596958 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g"] Nov 03 00:30:56.619968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.619940 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-cb65f3a5-dockercfg-bnxn7\"" Nov 03 00:30:56.646779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.646748 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qpfvs\" (UniqueName: \"kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs\") pod \"payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:56.686555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.686520 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94mgx\" (UniqueName: \"kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.686749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.686657 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.791697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.788556 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94mgx\" (UniqueName: \"kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.791697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.788672 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.805718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.799452 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.851730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.851652 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94mgx\" (UniqueName: \"kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx\") pod \"payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.889018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.888986 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:56.936499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:56.936462 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:57.160733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.160562 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx"] Nov 03 00:30:57.160918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.160830 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" containerID="cri-o://5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b" gracePeriod=30 Nov 03 00:30:57.215042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.214999 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x"] Nov 03 00:30:57.219976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.219950 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.277122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.277088 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g"] Nov 03 00:30:57.277625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.277542 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:30:57.277965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.277872 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:30:57.283954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.283839 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x"] Nov 03 00:30:57.303864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.303824 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.304037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.303944 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.304333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.304240 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22wh6\" (UniqueName: \"kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.380775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.380747 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw"] Nov 03 00:30:57.381250 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:57.381224 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c09bab9_c0d9_4845_9221_443c89cc7005.slice/crio-5eb0e7265afce6d0358df3f54db0f876d97ac899fefa835dd0aa36a013e73c49 WatchSource:0}: Error finding container 5eb0e7265afce6d0358df3f54db0f876d97ac899fefa835dd0aa36a013e73c49: Status 404 returned error can't find the container with id 5eb0e7265afce6d0358df3f54db0f876d97ac899fefa835dd0aa36a013e73c49 Nov 03 00:30:57.406136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.406105 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.406245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.406183 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.406515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.406497 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22wh6\" (UniqueName: \"kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.406865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.406844 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.408538 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.408520 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.424169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.424139 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" event={"ID":"7c09bab9-c0d9-4845-9221-443c89cc7005","Type":"ContainerStarted","Data":"5eb0e7265afce6d0358df3f54db0f876d97ac899fefa835dd0aa36a013e73c49"} Nov 03 00:30:57.425965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.425935 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" event={"ID":"5ba65ac5-5c99-4256-b9ba-05a111030b88","Type":"ContainerStarted","Data":"294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617"} Nov 03 00:30:57.426084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.425993 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:30:57.426084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.426015 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" event={"ID":"5ba65ac5-5c99-4256-b9ba-05a111030b88","Type":"ContainerStarted","Data":"b04f7f874d67ac701505f3a75550d1e180b71f0af099cc6650800da2c4b3e2a1"} Nov 03 00:30:57.463461 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.463435 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22wh6\" (UniqueName: \"kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6\") pod \"maas-api-revoke-sub-c3f14304-wzf6x\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.534667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.534612 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:30:57.538925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.538879 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" podStartSLOduration=1.538862948 podStartE2EDuration="1.538862948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:57.537900845 +0000 UTC m=+1547.683595303" watchObservedRunningTime="2026-11-03 00:30:57.538862948 +0000 UTC m=+1547.684557405" Nov 03 00:30:57.928730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:57.928690 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x"] Nov 03 00:30:57.929349 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:30:57.929324 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443a4a34_eb74_4615_a85f_bf35e4d9e6c7.slice/crio-b8d382576017eac54b75b04e047bc8ebc28639181a7df922aaa4fc65ac25ce18 WatchSource:0}: Error finding container b8d382576017eac54b75b04e047bc8ebc28639181a7df922aaa4fc65ac25ce18: Status 404 returned error can't find the container with id b8d382576017eac54b75b04e047bc8ebc28639181a7df922aaa4fc65ac25ce18 Nov 03 00:30:58.439057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.439021 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" event={"ID":"7c09bab9-c0d9-4845-9221-443c89cc7005","Type":"ContainerStarted","Data":"85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607"} Nov 03 00:30:58.439243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.439088 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:30:58.442138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.442104 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" event={"ID":"443a4a34-eb74-4615-a85f-bf35e4d9e6c7","Type":"ContainerStarted","Data":"e3bb2b4a89e009c91e0e2ef34c95096fce389a99b2b56c36a4879c4c69787dfd"} Nov 03 00:30:58.442288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.442159 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" event={"ID":"443a4a34-eb74-4615-a85f-bf35e4d9e6c7","Type":"ContainerStarted","Data":"b8d382576017eac54b75b04e047bc8ebc28639181a7df922aaa4fc65ac25ce18"} Nov 03 00:30:58.560888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.560838 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" podStartSLOduration=2.560825055 podStartE2EDuration="2.560825055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:58.560461838 +0000 UTC m=+1548.706156295" watchObservedRunningTime="2026-11-03 00:30:58.560825055 +0000 UTC m=+1548.706519529" Nov 03 00:30:58.692804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.692734 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" podStartSLOduration=1.69271467 podStartE2EDuration="1.69271467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:58.682130623 +0000 UTC m=+1548.827825078" watchObservedRunningTime="2026-11-03 00:30:58.69271467 +0000 UTC m=+1548.838409125" Nov 03 00:30:58.876820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.876793 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:30:58.954777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.954739 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.954844 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.954884 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.954930 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.954966 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955005 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955111 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955185 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955264 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token\") pod \"c360651c-0287-4473-9b64-8cbb2f8832d3\" (UID: \"c360651c-0287-4473-9b64-8cbb2f8832d3\") " Nov 03 00:30:58.955668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955277 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:58.955668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955293 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:58.955668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955373 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:30:58.955668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955377 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:58.955668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955547 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:58.956164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955856 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:58.956164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955878 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:58.956164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955887 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:58.956164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955895 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c360651c-0287-4473-9b64-8cbb2f8832d3-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:58.956164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.955906 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:58.957311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.957283 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:30:58.957415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.957344 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:30:58.957540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.957524 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "kube-api-access-468ts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:58.957768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:58.957753 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token" pod "c360651c-0287-4473-9b64-8cbb2f8832d3" (UID: "c360651c-0287-4473-9b64-8cbb2f8832d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:30:59.057692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.057655 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:59.057692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.057691 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-468ts\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-kube-api-access-468ts\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:59.057957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.057707 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:59.057957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.057718 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c360651c-0287-4473-9b64-8cbb2f8832d3-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:30:59.454780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.454739 5144 generic.go:356] "Generic (PLEG): container finished" podID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerID="26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51" exitCode=0 Nov 03 00:30:59.454965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.454806 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" Nov 03 00:30:59.454965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.454820 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" event={"ID":"c360651c-0287-4473-9b64-8cbb2f8832d3","Type":"ContainerDied","Data":"26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51"} Nov 03 00:30:59.454965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.454880 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq" event={"ID":"c360651c-0287-4473-9b64-8cbb2f8832d3","Type":"ContainerDied","Data":"d7db607bd24f899b927afb3a8122d31918fcff8c73c7621d34933047a0bf986f"} Nov 03 00:30:59.454965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.454904 5144 scope.go:122] "RemoveContainer" containerID="26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51" Nov 03 00:30:59.457232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.457201 5144 generic.go:356] "Generic (PLEG): container finished" podID="443a4a34-eb74-4615-a85f-bf35e4d9e6c7" containerID="e3bb2b4a89e009c91e0e2ef34c95096fce389a99b2b56c36a4879c4c69787dfd" exitCode=0 Nov 03 00:30:59.457347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.457286 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" event={"ID":"443a4a34-eb74-4615-a85f-bf35e4d9e6c7","Type":"ContainerDied","Data":"e3bb2b4a89e009c91e0e2ef34c95096fce389a99b2b56c36a4879c4c69787dfd"} Nov 03 00:30:59.468241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.468224 5144 scope.go:122] "RemoveContainer" containerID="297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c" Nov 03 00:30:59.479042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.479025 5144 scope.go:122] "RemoveContainer" containerID="26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51" Nov 03 00:30:59.479328 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:59.479304 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51\": container with ID starting with 26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51 not found: ID does not exist" containerID="26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51" Nov 03 00:30:59.479375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.479343 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51"} err="failed to get container status \"26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51\": rpc error: code = NotFound desc = could not find container \"26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51\": container with ID starting with 26792abd761510acd29d2ec23a9cc6fbcb6b2d7c033f7c095b3457e47ac12d51 not found: ID does not exist" Nov 03 00:30:59.479417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.479372 5144 scope.go:122] "RemoveContainer" containerID="297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c" Nov 03 00:30:59.479669 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:30:59.479620 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c\": container with ID starting with 297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c not found: ID does not exist" containerID="297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c" Nov 03 00:30:59.479732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.479678 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c"} err="failed to get container status \"297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c\": rpc error: code = NotFound desc = could not find container \"297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c\": container with ID starting with 297c16b8efe47772ae7fa56f653f67701b7140bcb8cea1e75b54f8883b21fe0c not found: ID does not exist" Nov 03 00:30:59.732529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.732448 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq"] Nov 03 00:30:59.736288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:30:59.736262 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-e671ea-openshift-default-86f69d6465-5pgxq"] Nov 03 00:31:00.001946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.001857 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" path="/var/lib/kubelet/pods/c360651c-0287-4473-9b64-8cbb2f8832d3/volumes" Nov 03 00:31:00.665598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.665568 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:31:00.784242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.784213 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca\") pod \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " Nov 03 00:31:00.784416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.784377 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca\") pod \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " Nov 03 00:31:00.784511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.784491 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6\" (UniqueName: \"kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6\") pod \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\" (UID: \"443a4a34-eb74-4615-a85f-bf35e4d9e6c7\") " Nov 03 00:31:00.784624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.784600 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca" pod "443a4a34-eb74-4615-a85f-bf35e4d9e6c7" (UID: "443a4a34-eb74-4615-a85f-bf35e4d9e6c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:00.785148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.785123 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:00.787045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.787015 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6" pod "443a4a34-eb74-4615-a85f-bf35e4d9e6c7" (UID: "443a4a34-eb74-4615-a85f-bf35e4d9e6c7"). InnerVolumeSpecName "kube-api-access-22wh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:00.787172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.787039 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca" pod "443a4a34-eb74-4615-a85f-bf35e4d9e6c7" (UID: "443a4a34-eb74-4615-a85f-bf35e4d9e6c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:31:00.886420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.886392 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:00.886420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:00.886417 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22wh6\" (UniqueName: \"kubernetes.io/projected/443a4a34-eb74-4615-a85f-bf35e4d9e6c7-kube-api-access-22wh6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:01.477570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:01.477544 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" Nov 03 00:31:01.478046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:01.477577 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x" event={"ID":"443a4a34-eb74-4615-a85f-bf35e4d9e6c7","Type":"ContainerDied","Data":"b8d382576017eac54b75b04e047bc8ebc28639181a7df922aaa4fc65ac25ce18"} Nov 03 00:31:01.478046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:01.477620 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d382576017eac54b75b04e047bc8ebc28639181a7df922aaa4fc65ac25ce18" Nov 03 00:31:02.460230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.460200 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:31:02.488614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.488576 5144 generic.go:356] "Generic (PLEG): container finished" podID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerID="5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b" exitCode=0 Nov 03 00:31:02.489043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.488649 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" Nov 03 00:31:02.489043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.488662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" event={"ID":"dbef4fbb-8a67-4440-a29e-f5b41ffd2321","Type":"ContainerDied","Data":"5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b"} Nov 03 00:31:02.489043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.488724 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx" event={"ID":"dbef4fbb-8a67-4440-a29e-f5b41ffd2321","Type":"ContainerDied","Data":"b769c95a95f8f29cb3dbb8f0dc1e45c107fdf9c04a16e7e8b735b84c2d03d080"} Nov 03 00:31:02.489043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.488746 5144 scope.go:122] "RemoveContainer" containerID="5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b" Nov 03 00:31:02.500403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.500381 5144 scope.go:122] "RemoveContainer" containerID="5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b" Nov 03 00:31:02.500723 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:02.500700 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b\": container with ID starting with 5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b not found: ID does not exist" containerID="5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b" Nov 03 00:31:02.500812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.500730 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b"} err="failed to get container status \"5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b\": rpc error: code = NotFound desc = could not find container \"5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b\": container with ID starting with 5d0f24e7cdf4303411c0c10d376da721945710fa6cdcaf9fe91b6cf16146a50b not found: ID does not exist" Nov 03 00:31:02.611006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.610971 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611056 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611107 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611266 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611313 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611300 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:02.611370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611347 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:02.611533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611361 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611410 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611459 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611466 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:02.611767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611546 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert\") pod \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\" (UID: \"dbef4fbb-8a67-4440-a29e-f5b41ffd2321\") " Nov 03 00:31:02.611767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611711 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:02.611944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.611922 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:02.612274 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.612253 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.612345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.612278 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.612345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.612291 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.612345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.612305 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.612345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.612318 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.613866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.613828 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "kube-api-access-vscbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:02.613983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.613915 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:02.614160 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.614132 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:02.614253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.614177 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token" pod "dbef4fbb-8a67-4440-a29e-f5b41ffd2321" (UID: "dbef4fbb-8a67-4440-a29e-f5b41ffd2321"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:02.713779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.713700 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vscbg\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-kube-api-access-vscbg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.713779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.713733 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.713779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.713745 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.713779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.713755 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbef4fbb-8a67-4440-a29e-f5b41ffd2321-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:02.911007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.910972 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx"] Nov 03 00:31:02.914807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:02.914781 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-k5clx"] Nov 03 00:31:03.997765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:03.997732 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" path="/var/lib/kubelet/pods/dbef4fbb-8a67-4440-a29e-f5b41ffd2321/volumes" Nov 03 00:31:04.825082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:04.825047 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps"] Nov 03 00:31:05.745320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.745288 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h"] Nov 03 00:31:05.746272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746250 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" Nov 03 00:31:05.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746274 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" Nov 03 00:31:05.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746303 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" Nov 03 00:31:05.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746310 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" Nov 03 00:31:05.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746399 5144 cpu_manager.go:414] "containerMap: removing container" podUID="443a4a34-eb74-4615-a85f-bf35e4d9e6c7" containerName="revoke-keys" Nov 03 00:31:05.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746406 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443a4a34-eb74-4615-a85f-bf35e4d9e6c7" containerName="revoke-keys" Nov 03 00:31:05.746738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746572 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbef4fbb-8a67-4440-a29e-f5b41ffd2321" containerName="istio-proxy" Nov 03 00:31:05.746738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746598 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="443a4a34-eb74-4615-a85f-bf35e4d9e6c7" containerName="revoke-keys" Nov 03 00:31:05.746738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746609 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" Nov 03 00:31:05.746738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.746622 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" Nov 03 00:31:05.751620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.751595 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.767714 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.767842 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.767886 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.767931 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.768019 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.768049 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.768085 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtzh4\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.768113 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.769097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.768140 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.818661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.806576 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-8c9581-openshift-default-dockercfg-gsnwc\"" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872393 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872518 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872559 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872601 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872704 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872733 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872770 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtzh4\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872802 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.872839 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.873330 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.873954 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.874666 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.875304 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.879693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.876286 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.884536 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.884474 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.887510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.887451 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.926920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.926858 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h"] Nov 03 00:31:05.942923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.942484 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtzh4\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:05.945170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:05.945117 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token\") pod \"e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:06.076455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:06.076424 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:06.555267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:06.555233 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h"] Nov 03 00:31:06.555795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:06.555767 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:06.555915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:06.555840 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:06.555915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:06.555886 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:07.542698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:07.542663 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" event={"ID":"f3b98f61-6633-4f03-b087-eb7645e39aba","Type":"ContainerStarted","Data":"d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820"} Nov 03 00:31:07.543195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:07.542728 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" event={"ID":"f3b98f61-6633-4f03-b087-eb7645e39aba","Type":"ContainerStarted","Data":"6b32dae6467c144dbedcc797a335bfaae7e35a5a6ecf20e8536263f9c6fdd5ff"} Nov 03 00:31:07.713037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:07.712957 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podStartSLOduration=2.712939723 podStartE2EDuration="2.712939723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:07.707736794 +0000 UTC m=+1557.853431252" watchObservedRunningTime="2026-11-03 00:31:07.712939723 +0000 UTC m=+1557.858634179" Nov 03 00:31:08.077986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:08.077944 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:08.084224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:08.084194 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:08.445941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:08.445857 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:31:08.552790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:08.552761 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:08.553813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:08.553792 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:31:09.460590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:09.460562 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:31:09.825667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:09.825613 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" podUID="d9ef3db3-cb62-41ba-b0d0-00251ecceff3" containerName="payload-pre-processing" containerID="cri-o://3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2" gracePeriod=25 Nov 03 00:31:09.918331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:09.918298 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5"] Nov 03 00:31:09.919073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:09.919052 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" Nov 03 00:31:09.919191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:09.919073 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c360651c-0287-4473-9b64-8cbb2f8832d3" containerName="istio-proxy" Nov 03 00:31:09.922940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:09.922915 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.133565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.133468 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3df441-openshift-default-dockercfg-twcmr\"" Nov 03 00:31:10.163870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.163831 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.163934 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.163998 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.164033 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.164091 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.164146 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.164235 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.164319 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.164427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.164370 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqftk\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.186529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.186485 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5"] Nov 03 00:31:10.265451 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.265417 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.265756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.265738 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.265944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.265927 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266087 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqftk\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266235 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266172 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266489 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266578 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266666 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266704 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266739 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266759 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.266994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.266918 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.267398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.267379 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.275272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.269292 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.275272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.269850 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.323406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.323373 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.323590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.323534 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqftk\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk\") pod \"e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.412506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.412424 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:31:10.538710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.538672 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:10.570781 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.570746 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume\") pod \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " Nov 03 00:31:10.571076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.571056 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt\" (UniqueName: \"kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt\") pod \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\" (UID: \"d9ef3db3-cb62-41ba-b0d0-00251ecceff3\") " Nov 03 00:31:10.574082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.574049 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume" pod "d9ef3db3-cb62-41ba-b0d0-00251ecceff3" (UID: "d9ef3db3-cb62-41ba-b0d0-00251ecceff3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:10.576727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.576693 5144 generic.go:356] "Generic (PLEG): container finished" podID="d9ef3db3-cb62-41ba-b0d0-00251ecceff3" containerID="3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2" exitCode=0 Nov 03 00:31:10.577265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.577202 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" Nov 03 00:31:10.577736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.577617 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" event={"ID":"d9ef3db3-cb62-41ba-b0d0-00251ecceff3","Type":"ContainerDied","Data":"3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2"} Nov 03 00:31:10.577736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.577704 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps" event={"ID":"d9ef3db3-cb62-41ba-b0d0-00251ecceff3","Type":"ContainerDied","Data":"445d44de49bff96af2f9198b971a03c8cc761a564ce6029f691118f81628b4b9"} Nov 03 00:31:10.577736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.577739 5144 scope.go:122] "RemoveContainer" containerID="3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2" Nov 03 00:31:10.584952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.584919 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt" pod "d9ef3db3-cb62-41ba-b0d0-00251ecceff3" (UID: "d9ef3db3-cb62-41ba-b0d0-00251ecceff3"). InnerVolumeSpecName "kube-api-access-9tlrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:10.595189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.595163 5144 scope.go:122] "RemoveContainer" containerID="3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2" Nov 03 00:31:10.595573 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:10.595551 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2\": container with ID starting with 3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2 not found: ID does not exist" containerID="3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2" Nov 03 00:31:10.595664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.595582 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2"} err="failed to get container status \"3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2\": rpc error: code = NotFound desc = could not find container \"3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2\": container with ID starting with 3f64088ff619dcf00b023ce8e9a42e9994e2ead5fc3a5f1de7bf790d87246ff2 not found: ID does not exist" Nov 03 00:31:10.673941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.673855 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:10.673941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.673888 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tlrt\" (UniqueName: \"kubernetes.io/projected/d9ef3db3-cb62-41ba-b0d0-00251ecceff3-kube-api-access-9tlrt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:10.998234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:10.998145 5144 scope.go:122] "RemoveContainer" containerID="b9a103ba4a3977e56f964f02105c9193410782e3f990c1c061f7caf34dc71497" Nov 03 00:31:11.010105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.010078 5144 scope.go:122] "RemoveContainer" containerID="87fcbc8b19298d029da56362bc414f84e7cb24f52c489382b4f4d63de358fe7b" Nov 03 00:31:11.013704 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:31:11.013675 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f02589b_9510_4e1e_9535_42cda380ff19.slice/crio-ea67fb3db94533d21a4600e399dc177c1ba905a816ab22a4ae0e5203abf8c45b WatchSource:0}: Error finding container ea67fb3db94533d21a4600e399dc177c1ba905a816ab22a4ae0e5203abf8c45b: Status 404 returned error can't find the container with id ea67fb3db94533d21a4600e399dc177c1ba905a816ab22a4ae0e5203abf8c45b Nov 03 00:31:11.014818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.014736 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5"] Nov 03 00:31:11.017082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.017047 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:11.017172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.017145 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:11.017228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.017187 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:31:11.019036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.019012 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps"] Nov 03 00:31:11.027362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.027340 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-38f5a3-8495cdc7d7-hzbps"] Nov 03 00:31:11.041443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.041421 5144 scope.go:122] "RemoveContainer" containerID="f99af73b9d895009340895aa25aaf9432f2c6d3746f1877ad9dbcc76064f33b4" Nov 03 00:31:11.055485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.055462 5144 scope.go:122] "RemoveContainer" containerID="193d350b7c375dbbb5326a9c37f8a189406a771c2fbd6be0e5c0a41a363659e2" Nov 03 00:31:11.069696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.069668 5144 scope.go:122] "RemoveContainer" containerID="c1664286bc052cfb8a535305cb3ca9ce468c543468463f754bf5149a93c37885" Nov 03 00:31:11.082309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.082285 5144 scope.go:122] "RemoveContainer" containerID="7fa7a2875679881686a58e8f0db00f6bcdae8309611e13bf3c498c29ec9b48e5" Nov 03 00:31:11.099817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.099790 5144 scope.go:122] "RemoveContainer" containerID="c5ca7c13564712f582491ad5d7df5a7761db4cfefdf820ec448f8d91d3c72b47" Nov 03 00:31:11.113766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.113741 5144 scope.go:122] "RemoveContainer" containerID="189d0362c419ca4d20bf05c461c015a3561d808ec715823fcdbd9e9dd487e277" Nov 03 00:31:11.125287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.125260 5144 scope.go:122] "RemoveContainer" containerID="74eb0efe88d75ab40cad020d5f6a4ace71be35bd756b537e65b581d12c213ac6" Nov 03 00:31:11.589662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.589545 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" event={"ID":"3f02589b-9510-4e1e-9535-42cda380ff19","Type":"ContainerStarted","Data":"930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2"} Nov 03 00:31:11.589662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.589604 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" event={"ID":"3f02589b-9510-4e1e-9535-42cda380ff19","Type":"ContainerStarted","Data":"ea67fb3db94533d21a4600e399dc177c1ba905a816ab22a4ae0e5203abf8c45b"} Nov 03 00:31:11.710450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:11.710385 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podStartSLOduration=2.710371486 podStartE2EDuration="2.710371486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:11.708502392 +0000 UTC m=+1561.854196865" watchObservedRunningTime="2026-11-03 00:31:11.710371486 +0000 UTC m=+1561.856065943" Nov 03 00:31:12.000720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:12.000620 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9ef3db3-cb62-41ba-b0d0-00251ecceff3" path="/var/lib/kubelet/pods/d9ef3db3-cb62-41ba-b0d0-00251ecceff3/volumes" Nov 03 00:31:12.157874 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:12.157832 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:31:12.539682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:12.539613 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:12.545271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:12.545244 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:12.598767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:12.598731 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:12.599926 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:12.599905 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:31:14.127681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.127608 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q"] Nov 03 00:31:14.128307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.128291 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d9ef3db3-cb62-41ba-b0d0-00251ecceff3" containerName="payload-pre-processing" Nov 03 00:31:14.128371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.128307 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9ef3db3-cb62-41ba-b0d0-00251ecceff3" containerName="payload-pre-processing" Nov 03 00:31:14.128455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.128444 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9ef3db3-cb62-41ba-b0d0-00251ecceff3" containerName="payload-pre-processing" Nov 03 00:31:14.134413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.134389 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.138571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.138538 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.138749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.138731 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjw2s\" (UniqueName: \"kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.187828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.187795 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q"] Nov 03 00:31:14.239678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.235966 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8c9581\"" Nov 03 00:31:14.239678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.236721 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8c9581-dockercfg-svkz2\"" Nov 03 00:31:14.271657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.265616 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.272096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.272071 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjw2s\" (UniqueName: \"kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.293659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.274808 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.343652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.343343 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb"] Nov 03 00:31:14.392075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.391995 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjw2s\" (UniqueName: \"kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s\") pod \"payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.402907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.402307 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb"] Nov 03 00:31:14.402907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.402438 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.450534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.450496 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:14.484704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.483507 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.484704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.483719 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcpxw\" (UniqueName: \"kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw\") pod \"payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.585028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.584990 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcpxw\" (UniqueName: \"kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw\") pod \"payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.585226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.585202 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.585996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.585974 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.642737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.642619 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcpxw\" (UniqueName: \"kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw\") pod \"payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.724428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.724396 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:14.834147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:14.833163 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q"] Nov 03 00:31:15.106175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.106153 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb"] Nov 03 00:31:15.106865 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:31:15.106836 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95d709e4_4ce5_4fda_ada1_92775f3315e1.slice/crio-3b3111582a5cc4bfccf56fcc08bdbb36dab75b9cc99bf4b258b31b25ffabed70 WatchSource:0}: Error finding container 3b3111582a5cc4bfccf56fcc08bdbb36dab75b9cc99bf4b258b31b25ffabed70: Status 404 returned error can't find the container with id 3b3111582a5cc4bfccf56fcc08bdbb36dab75b9cc99bf4b258b31b25ffabed70 Nov 03 00:31:15.627230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.627189 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" event={"ID":"56252b9f-c05f-4a2b-b615-18587c611828","Type":"ContainerStarted","Data":"b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6"} Nov 03 00:31:15.627766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.627255 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" event={"ID":"56252b9f-c05f-4a2b-b615-18587c611828","Type":"ContainerStarted","Data":"c0b243498d6fd82455e021a1216d22e5ea353ae9bd2e0257aaf2bda20358d03f"} Nov 03 00:31:15.627766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.627352 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:15.629876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.629844 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" event={"ID":"95d709e4-4ce5-4fda-ada1-92775f3315e1","Type":"ContainerStarted","Data":"5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f"} Nov 03 00:31:15.629987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.629892 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" event={"ID":"95d709e4-4ce5-4fda-ada1-92775f3315e1","Type":"ContainerStarted","Data":"3b3111582a5cc4bfccf56fcc08bdbb36dab75b9cc99bf4b258b31b25ffabed70"} Nov 03 00:31:15.629987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.629920 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:15.743902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.743842 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" podStartSLOduration=1.743827835 podStartE2EDuration="1.743827835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:15.742896741 +0000 UTC m=+1565.888591244" watchObservedRunningTime="2026-11-03 00:31:15.743827835 +0000 UTC m=+1565.889522335" Nov 03 00:31:15.861410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:15.861306 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" podStartSLOduration=1.8612665910000001 podStartE2EDuration="1.861266591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:15.860776869 +0000 UTC m=+1566.006471327" watchObservedRunningTime="2026-11-03 00:31:15.861266591 +0000 UTC m=+1566.006961050" Nov 03 00:31:19.819382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:19.819154 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp"] Nov 03 00:31:23.980973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:23.980938 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn"] Nov 03 00:31:23.985586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:23.985568 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.041932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.041900 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn"] Nov 03 00:31:24.109656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.107084 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-3df441-dockercfg-4q7mm\"" Nov 03 00:31:24.109656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.107177 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-3df441\"" Nov 03 00:31:24.213695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.200685 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stt2\" (UniqueName: \"kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2\") pod \"payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.213695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.200885 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.302421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.302375 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.302620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.302538 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stt2\" (UniqueName: \"kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2\") pod \"payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.303064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.303041 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.359493 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.359452 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stt2\" (UniqueName: \"kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2\") pod \"payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.609255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.609140 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:24.820340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.820264 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" podUID="694a220a-db40-4d48-a822-b2d70894efd3" containerName="payload-pre-processing" containerID="cri-o://1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009" gracePeriod=25 Nov 03 00:31:24.996437 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:24.996405 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn"] Nov 03 00:31:25.000883 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:31:25.000848 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6905a5fa_c3e0_4904_8021_23503d9dc816.slice/crio-79f1bff355fd091e9d80328d4adb39b91d02d115afe29a1ba1e528aa8a057947 WatchSource:0}: Error finding container 79f1bff355fd091e9d80328d4adb39b91d02d115afe29a1ba1e528aa8a057947: Status 404 returned error can't find the container with id 79f1bff355fd091e9d80328d4adb39b91d02d115afe29a1ba1e528aa8a057947 Nov 03 00:31:25.101362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.101057 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:31:25.123085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.122383 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume\") pod \"694a220a-db40-4d48-a822-b2d70894efd3\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " Nov 03 00:31:25.123085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.122448 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt\" (UniqueName: \"kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt\") pod \"694a220a-db40-4d48-a822-b2d70894efd3\" (UID: \"694a220a-db40-4d48-a822-b2d70894efd3\") " Nov 03 00:31:25.123585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.123558 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume" pod "694a220a-db40-4d48-a822-b2d70894efd3" (UID: "694a220a-db40-4d48-a822-b2d70894efd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:25.125327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.125302 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt" pod "694a220a-db40-4d48-a822-b2d70894efd3" (UID: "694a220a-db40-4d48-a822-b2d70894efd3"). InnerVolumeSpecName "kube-api-access-j2cvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:25.224853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.224774 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/694a220a-db40-4d48-a822-b2d70894efd3-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:25.224853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.224811 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2cvt\" (UniqueName: \"kubernetes.io/projected/694a220a-db40-4d48-a822-b2d70894efd3-kube-api-access-j2cvt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:25.802428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.802388 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" event={"ID":"6905a5fa-c3e0-4904-8021-23503d9dc816","Type":"ContainerStarted","Data":"efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768"} Nov 03 00:31:25.802668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.802456 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" event={"ID":"6905a5fa-c3e0-4904-8021-23503d9dc816","Type":"ContainerStarted","Data":"79f1bff355fd091e9d80328d4adb39b91d02d115afe29a1ba1e528aa8a057947"} Nov 03 00:31:25.802668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.802486 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:25.804024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.803997 5144 generic.go:356] "Generic (PLEG): container finished" podID="694a220a-db40-4d48-a822-b2d70894efd3" containerID="1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009" exitCode=0 Nov 03 00:31:25.804164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.804034 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" event={"ID":"694a220a-db40-4d48-a822-b2d70894efd3","Type":"ContainerDied","Data":"1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009"} Nov 03 00:31:25.804164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.804079 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" event={"ID":"694a220a-db40-4d48-a822-b2d70894efd3","Type":"ContainerDied","Data":"1c6a79ad8385bd96c833fe4ee75f7238954b55088f7e33966cf5b74239453a49"} Nov 03 00:31:25.804164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.804107 5144 scope.go:122] "RemoveContainer" containerID="1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009" Nov 03 00:31:25.804164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.804037 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp" Nov 03 00:31:25.815861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.815833 5144 scope.go:122] "RemoveContainer" containerID="1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009" Nov 03 00:31:25.816188 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:25.816165 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009\": container with ID starting with 1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009 not found: ID does not exist" containerID="1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009" Nov 03 00:31:25.816250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.816199 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009"} err="failed to get container status \"1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009\": rpc error: code = NotFound desc = could not find container \"1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009\": container with ID starting with 1134696b8df2b554cb04ce919ca17b7f0c40759a772d8dcf6380fe97db51b009 not found: ID does not exist" Nov 03 00:31:25.918730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:25.918669 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" podStartSLOduration=2.918654948 podStartE2EDuration="2.918654948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:25.91601148 +0000 UTC m=+1576.061705958" watchObservedRunningTime="2026-11-03 00:31:25.918654948 +0000 UTC m=+1576.064349419" Nov 03 00:31:26.036486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:26.036440 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp"] Nov 03 00:31:26.046868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:26.046832 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-be578d-5f69f7d7f9-jn7kp"] Nov 03 00:31:26.642352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:26.642313 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:31:26.642730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:26.642708 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:31:28.001887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:28.001829 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694a220a-db40-4d48-a822-b2d70894efd3" path="/var/lib/kubelet/pods/694a220a-db40-4d48-a822-b2d70894efd3/volumes" Nov 03 00:31:28.670735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:28.670702 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g"] Nov 03 00:31:28.670963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:28.670938 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" containerID="cri-o://2afe78bcbd3d816a803e4fd8abcae9d112a7b28a965083e177189c7ef0cf8770" gracePeriod=30 Nov 03 00:31:33.880528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:33.880491 5144 generic.go:356] "Generic (PLEG): container finished" podID="d418e880-f0d2-466a-8761-a2f1328d587e" containerID="2afe78bcbd3d816a803e4fd8abcae9d112a7b28a965083e177189c7ef0cf8770" exitCode=0 Nov 03 00:31:33.881224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:33.881189 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" event={"ID":"d418e880-f0d2-466a-8761-a2f1328d587e","Type":"ContainerDied","Data":"2afe78bcbd3d816a803e4fd8abcae9d112a7b28a965083e177189c7ef0cf8770"} Nov 03 00:31:33.963434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:33.963395 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:31:34.064979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.064944 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065019 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065054 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065161 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065201 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065265 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065386 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065435 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.065562 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065497 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo\") pod \"d418e880-f0d2-466a-8761-a2f1328d587e\" (UID: \"d418e880-f0d2-466a-8761-a2f1328d587e\") " Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065780 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065881 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065917 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.065994 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.066009 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.066622 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.066696 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.066710 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.066724 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.066883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.066737 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d418e880-f0d2-466a-8761-a2f1328d587e-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.067966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.067935 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "kube-api-access-n6ksc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.068115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.068092 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:34.068196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.068120 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:34.068438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.068408 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo" pod "d418e880-f0d2-466a-8761-a2f1328d587e" (UID: "d418e880-f0d2-466a-8761-a2f1328d587e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:34.168922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.168864 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.168922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.168906 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6ksc\" (UniqueName: \"kubernetes.io/projected/d418e880-f0d2-466a-8761-a2f1328d587e-kube-api-access-n6ksc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.168922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.168924 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d418e880-f0d2-466a-8761-a2f1328d587e-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.169325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.168939 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d418e880-f0d2-466a-8761-a2f1328d587e-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:34.892074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.892043 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" Nov 03 00:31:34.892524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.892036 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g" event={"ID":"d418e880-f0d2-466a-8761-a2f1328d587e","Type":"ContainerDied","Data":"e8325ca6d9dc807f84b4296b0a745aa57f25e93962d765401d514c8a3c47b3e2"} Nov 03 00:31:34.892524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:34.892206 5144 scope.go:122] "RemoveContainer" containerID="2afe78bcbd3d816a803e4fd8abcae9d112a7b28a965083e177189c7ef0cf8770" Nov 03 00:31:35.107370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:35.107331 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g"] Nov 03 00:31:35.157371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:35.157259 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-jt58g"] Nov 03 00:31:35.999303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:35.999267 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" path="/var/lib/kubelet/pods/d418e880-f0d2-466a-8761-a2f1328d587e/volumes" Nov 03 00:31:36.818013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:36.817978 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:31:42.158031 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:42.157996 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:31:49.821019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:49.820982 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg"] Nov 03 00:31:49.821454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:49.821209 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" containerID="cri-o://192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed" gracePeriod=30 Nov 03 00:31:55.193674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.193621 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:31:55.212152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.212117 5144 generic.go:356] "Generic (PLEG): container finished" podID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerID="192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed" exitCode=0 Nov 03 00:31:55.212347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.212177 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" Nov 03 00:31:55.212347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.212197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" event={"ID":"363102b0-1d5d-4235-8a0f-c7acca33b428","Type":"ContainerDied","Data":"192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed"} Nov 03 00:31:55.212347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.212251 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg" event={"ID":"363102b0-1d5d-4235-8a0f-c7acca33b428","Type":"ContainerDied","Data":"4a9d792c1a18e3f205b24bb37164d21604b749d15497ce4fcd9dfc91f353325a"} Nov 03 00:31:55.212347 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.212262 5144 scope.go:122] "RemoveContainer" containerID="192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed" Nov 03 00:31:55.247175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.247153 5144 scope.go:122] "RemoveContainer" containerID="bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47" Nov 03 00:31:55.255858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.255826 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.255889 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.255949 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256036 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256070 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256159 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256114 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256151 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:55.256231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256211 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256260 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256336 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert\") pod \"363102b0-1d5d-4235-8a0f-c7acca33b428\" (UID: \"363102b0-1d5d-4235-8a0f-c7acca33b428\") " Nov 03 00:31:55.256476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256384 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:55.256815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256579 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:55.256815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256766 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:55.256946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256870 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.256946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256883 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.256946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256892 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.256946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256901 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.256946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.256912 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:31:55.259036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.259000 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:31:55.259151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.259043 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:55.259219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.259201 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:31:55.260034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.260014 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft" pod "363102b0-1d5d-4235-8a0f-c7acca33b428" (UID: "363102b0-1d5d-4235-8a0f-c7acca33b428"). InnerVolumeSpecName "kube-api-access-tcmft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:31:55.260400 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.260386 5144 scope.go:122] "RemoveContainer" containerID="192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed" Nov 03 00:31:55.260857 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:55.260834 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed\": container with ID starting with 192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed not found: ID does not exist" containerID="192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed" Nov 03 00:31:55.260909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.260869 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed"} err="failed to get container status \"192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed\": rpc error: code = NotFound desc = could not find container \"192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed\": container with ID starting with 192a50d443ef8cd3bf6eab7161b217e1dc4dbd8dd0d5e5256e9a6e99183de1ed not found: ID does not exist" Nov 03 00:31:55.260909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.260888 5144 scope.go:122] "RemoveContainer" containerID="bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47" Nov 03 00:31:55.261170 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:31:55.261142 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47\": container with ID starting with bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47 not found: ID does not exist" containerID="bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47" Nov 03 00:31:55.261214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.261169 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47"} err="failed to get container status \"bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47\": rpc error: code = NotFound desc = could not find container \"bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47\": container with ID starting with bd4b2df4f7cac4850e93c4d9c36692ab4e1da9fafd8e269e9a3bc02c5dc27f47 not found: ID does not exist" Nov 03 00:31:55.358529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.358484 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcmft\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-kube-api-access-tcmft\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.358529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.358520 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.358529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.358530 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363102b0-1d5d-4235-8a0f-c7acca33b428-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.358529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.358539 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.358862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.358549 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363102b0-1d5d-4235-8a0f-c7acca33b428-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:31:55.665682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.665639 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg"] Nov 03 00:31:55.672233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.672207 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-be578d-openshift-default-69787f56bc-wzzmg"] Nov 03 00:31:55.998967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:55.998882 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" path="/var/lib/kubelet/pods/363102b0-1d5d-4235-8a0f-c7acca33b428/volumes" Nov 03 00:31:57.391501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:31:57.391470 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl"] Nov 03 00:32:02.392592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.392547 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" podUID="86f3df52-ecd6-479e-9569-583495653896" containerName="payload-pre-processing" containerID="cri-o://7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78" gracePeriod=25 Nov 03 00:32:02.682448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.682423 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:32:02.825804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.825760 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume\") pod \"86f3df52-ecd6-479e-9569-583495653896\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " Nov 03 00:32:02.826002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.825845 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2\" (UniqueName: \"kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2\") pod \"86f3df52-ecd6-479e-9569-583495653896\" (UID: \"86f3df52-ecd6-479e-9569-583495653896\") " Nov 03 00:32:02.826276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.826247 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume" pod "86f3df52-ecd6-479e-9569-583495653896" (UID: "86f3df52-ecd6-479e-9569-583495653896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:02.826950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.826919 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86f3df52-ecd6-479e-9569-583495653896-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:02.829034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.829004 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2" pod "86f3df52-ecd6-479e-9569-583495653896" (UID: "86f3df52-ecd6-479e-9569-583495653896"). InnerVolumeSpecName "kube-api-access-pdwd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:02.929027 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:02.928931 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdwd2\" (UniqueName: \"kubernetes.io/projected/86f3df52-ecd6-479e-9569-583495653896-kube-api-access-pdwd2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:03.316318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.316271 5144 generic.go:356] "Generic (PLEG): container finished" podID="86f3df52-ecd6-479e-9569-583495653896" containerID="7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78" exitCode=0 Nov 03 00:32:03.316510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.316387 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" Nov 03 00:32:03.316510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.316412 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" event={"ID":"86f3df52-ecd6-479e-9569-583495653896","Type":"ContainerDied","Data":"7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78"} Nov 03 00:32:03.316510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.316468 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl" event={"ID":"86f3df52-ecd6-479e-9569-583495653896","Type":"ContainerDied","Data":"db4a9f8a082fbf0261ad70e639edcafc206debdb9e39246189a1b36a19394326"} Nov 03 00:32:03.316510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.316506 5144 scope.go:122] "RemoveContainer" containerID="7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78" Nov 03 00:32:03.339467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.339444 5144 scope.go:122] "RemoveContainer" containerID="7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78" Nov 03 00:32:03.339891 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:32:03.339869 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78\": container with ID starting with 7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78 not found: ID does not exist" containerID="7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78" Nov 03 00:32:03.339972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.339903 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78"} err="failed to get container status \"7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78\": rpc error: code = NotFound desc = could not find container \"7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78\": container with ID starting with 7b98e6b6f443964df9e8dcf47e68fd563346648d0067cf0af06e3a3ef3d6fa78 not found: ID does not exist" Nov 03 00:32:03.480181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.480145 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl"] Nov 03 00:32:03.532021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.531977 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-c521ba-6fd8596cc9-69xfl"] Nov 03 00:32:03.998709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:03.998676 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86f3df52-ecd6-479e-9569-583495653896" path="/var/lib/kubelet/pods/86f3df52-ecd6-479e-9569-583495653896/volumes" Nov 03 00:32:11.371718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:11.371686 5144 scope.go:122] "RemoveContainer" containerID="a98e65dee02a0e82cefe2fb6a90b95ac338815ae4d1e6b468368c3887d1d52e7" Nov 03 00:32:11.382934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:11.382911 5144 scope.go:122] "RemoveContainer" containerID="bc043dbf7aefb6c58a40bd6cdeb189c802aeba59360e739e1f2e53506282bfd4" Nov 03 00:32:11.394537 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:11.394515 5144 scope.go:122] "RemoveContainer" containerID="e822ffb7e1618672fb24c309bdcf7443743f26fbc0a7b6e1566941712dafa2ab" Nov 03 00:32:11.403664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:11.403614 5144 scope.go:122] "RemoveContainer" containerID="9479e4366b699bd9c5ee220a90205d8988c8805133e41a1cac6e05afe6644a27" Nov 03 00:32:12.165131 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:32:12.165089 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:32:26.139847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.135490 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b"] Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142786 5144 cpu_manager.go:414] "containerMap: removing container" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142825 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142854 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142862 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142893 5144 cpu_manager.go:414] "containerMap: removing container" podUID="694a220a-db40-4d48-a822-b2d70894efd3" containerName="payload-pre-processing" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142901 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694a220a-db40-4d48-a822-b2d70894efd3" containerName="payload-pre-processing" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142921 5144 cpu_manager.go:414] "containerMap: removing container" podUID="86f3df52-ecd6-479e-9569-583495653896" containerName="payload-pre-processing" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.142929 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86f3df52-ecd6-479e-9569-583495653896" containerName="payload-pre-processing" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.143129 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.143144 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d418e880-f0d2-466a-8761-a2f1328d587e" containerName="istio-proxy" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.143154 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="694a220a-db40-4d48-a822-b2d70894efd3" containerName="payload-pre-processing" Nov 03 00:32:26.144653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.143165 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="86f3df52-ecd6-479e-9569-583495653896" containerName="payload-pre-processing" Nov 03 00:32:26.152023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.152000 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.236735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.236683 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql75d\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.237164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.237139 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.237395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.237378 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.237670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.237623 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.237886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.237848 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.238073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.238040 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.238275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.238258 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.238502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.238484 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.238677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.238662 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.263683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.261740 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b"] Nov 03 00:32:26.345727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.345514 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.345727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.345573 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346032 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346075 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346103 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346181 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346269 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346314 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346406 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql75d\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.347165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.346874 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.348649 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.348584 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.348915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.348886 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.349128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.349107 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.349187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.349168 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.350516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.350495 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.351028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.351007 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.403068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.402996 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.403224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.403178 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql75d\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d\") pod \"e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.470787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.470747 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:26.995855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.995827 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b"] Nov 03 00:32:26.995963 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:32:26.995883 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb487d783_1665_4742_ab58_ac8ed1cb2b43.slice/crio-51c86fff75997ff3e7f85200db147a42b43ecb876dbe094f47868d7a97ddfc30 WatchSource:0}: Error finding container 51c86fff75997ff3e7f85200db147a42b43ecb876dbe094f47868d7a97ddfc30: Status 404 returned error can't find the container with id 51c86fff75997ff3e7f85200db147a42b43ecb876dbe094f47868d7a97ddfc30 Nov 03 00:32:26.998135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.998105 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:26.998218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.998166 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:26.998218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:26.998194 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:27.677094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:27.677006 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" event={"ID":"b487d783-1665-4742-ab58-ac8ed1cb2b43","Type":"ContainerStarted","Data":"d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57"} Nov 03 00:32:27.677094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:27.677081 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" event={"ID":"b487d783-1665-4742-ab58-ac8ed1cb2b43","Type":"ContainerStarted","Data":"51c86fff75997ff3e7f85200db147a42b43ecb876dbe094f47868d7a97ddfc30"} Nov 03 00:32:27.846825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:27.846752 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podStartSLOduration=1.846730765 podStartE2EDuration="1.846730765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:27.842080058 +0000 UTC m=+1637.987774546" watchObservedRunningTime="2026-11-03 00:32:27.846730765 +0000 UTC m=+1637.992425226" Nov 03 00:32:28.471441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:28.471405 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:28.472788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:28.472761 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 00:32:28.472902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:28.472814 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 00:32:28.549747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:28.546686 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn"] Nov 03 00:32:28.646661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:28.644672 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs"] Nov 03 00:32:29.471593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:29.471554 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 00:32:29.472099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:29.471678 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 00:32:30.473722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:30.473049 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 00:32:30.473722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:30.473112 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 00:32:31.471445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:31.471407 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 00:32:31.471617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:31.471479 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 00:32:32.472115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:32.472075 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 00:32:32.472609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:32.472152 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 00:32:33.471119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.471088 5144 patch_prober.go:29] interesting pod/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 00:32:33.471378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.471141 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 00:32:33.547385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.547348 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" podUID="678b16e7-6c48-4246-81ae-134a663e6046" containerName="payload-processing" containerID="cri-o://3fdce939e8fd90b258b76c9fb06c3c0c192e0e7192d8511beb61ef7ee28bbaf8" gracePeriod=25 Nov 03 00:32:33.645282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.645248 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" podUID="cbd23a01-4668-4afb-8116-f35c52087662" containerName="payload-pre-processing" containerID="cri-o://f40d1b1934b41ac3240c318df3ca9667e415e9b3d56cc7baa954b29a13fca062" gracePeriod=25 Nov 03 00:32:33.777315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.777259 5144 generic.go:356] "Generic (PLEG): container finished" podID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerID="d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820" exitCode=0 Nov 03 00:32:33.777652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.777614 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" event={"ID":"f3b98f61-6633-4f03-b087-eb7645e39aba","Type":"ContainerDied","Data":"d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820"} Nov 03 00:32:33.778200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.778183 5144 scope.go:122] "RemoveContainer" containerID="d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820" Nov 03 00:32:33.788975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.780729 5144 generic.go:356] "Generic (PLEG): container finished" podID="cbd23a01-4668-4afb-8116-f35c52087662" containerID="f40d1b1934b41ac3240c318df3ca9667e415e9b3d56cc7baa954b29a13fca062" exitCode=0 Nov 03 00:32:33.788975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.781157 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" event={"ID":"cbd23a01-4668-4afb-8116-f35c52087662","Type":"ContainerDied","Data":"f40d1b1934b41ac3240c318df3ca9667e415e9b3d56cc7baa954b29a13fca062"} Nov 03 00:32:33.791982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.791957 5144 generic.go:356] "Generic (PLEG): container finished" podID="678b16e7-6c48-4246-81ae-134a663e6046" containerID="3fdce939e8fd90b258b76c9fb06c3c0c192e0e7192d8511beb61ef7ee28bbaf8" exitCode=0 Nov 03 00:32:33.792088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.791995 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" event={"ID":"678b16e7-6c48-4246-81ae-134a663e6046","Type":"ContainerDied","Data":"3fdce939e8fd90b258b76c9fb06c3c0c192e0e7192d8511beb61ef7ee28bbaf8"} Nov 03 00:32:33.799453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.798555 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:33.799453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.798663 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:33.799453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.798711 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:32:33.909439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:33.906978 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:32:34.004703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.004080 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf\" (UniqueName: \"kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf\") pod \"678b16e7-6c48-4246-81ae-134a663e6046\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " Nov 03 00:32:34.004703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.004245 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume\") pod \"678b16e7-6c48-4246-81ae-134a663e6046\" (UID: \"678b16e7-6c48-4246-81ae-134a663e6046\") " Nov 03 00:32:34.008413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.008380 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf" pod "678b16e7-6c48-4246-81ae-134a663e6046" (UID: "678b16e7-6c48-4246-81ae-134a663e6046"). InnerVolumeSpecName "kube-api-access-n2qqf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:34.014851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.014817 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume" pod "678b16e7-6c48-4246-81ae-134a663e6046" (UID: "678b16e7-6c48-4246-81ae-134a663e6046"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:34.035572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.035488 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:32:34.105773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.105739 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7\" (UniqueName: \"kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7\") pod \"cbd23a01-4668-4afb-8116-f35c52087662\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " Nov 03 00:32:34.106321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.106020 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume\") pod \"cbd23a01-4668-4afb-8116-f35c52087662\" (UID: \"cbd23a01-4668-4afb-8116-f35c52087662\") " Nov 03 00:32:34.106969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.106710 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2qqf\" (UniqueName: \"kubernetes.io/projected/678b16e7-6c48-4246-81ae-134a663e6046-kube-api-access-n2qqf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:34.106969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.106738 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/678b16e7-6c48-4246-81ae-134a663e6046-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:34.106969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.106925 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume" pod "cbd23a01-4668-4afb-8116-f35c52087662" (UID: "cbd23a01-4668-4afb-8116-f35c52087662"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:32:34.109476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.108908 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7" pod "cbd23a01-4668-4afb-8116-f35c52087662" (UID: "cbd23a01-4668-4afb-8116-f35c52087662"). InnerVolumeSpecName "kube-api-access-hjbr7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:34.208388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.208343 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbd23a01-4668-4afb-8116-f35c52087662-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:34.208388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.208379 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjbr7\" (UniqueName: \"kubernetes.io/projected/cbd23a01-4668-4afb-8116-f35c52087662-kube-api-access-hjbr7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:34.475270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.475192 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:34.475511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.475491 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:34.476285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.476260 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:32:34.802409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.802366 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" event={"ID":"cbd23a01-4668-4afb-8116-f35c52087662","Type":"ContainerDied","Data":"a3347f8eb3fdb63412b3b5ca5885cbb49794100d5281f59dfa3113bdc45eb762"} Nov 03 00:32:34.802898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.802471 5144 scope.go:122] "RemoveContainer" containerID="f40d1b1934b41ac3240c318df3ca9667e415e9b3d56cc7baa954b29a13fca062" Nov 03 00:32:34.802898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.802680 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs" Nov 03 00:32:34.805227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.805200 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" event={"ID":"678b16e7-6c48-4246-81ae-134a663e6046","Type":"ContainerDied","Data":"9bc29372a39ddc675a25e9b7809be6a7885ae3b3f7a55258c436c027d8433cd9"} Nov 03 00:32:34.805343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.805329 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn" Nov 03 00:32:34.856923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.856892 5144 scope.go:122] "RemoveContainer" containerID="3fdce939e8fd90b258b76c9fb06c3c0c192e0e7192d8511beb61ef7ee28bbaf8" Nov 03 00:32:34.857221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.857142 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" event={"ID":"f3b98f61-6633-4f03-b087-eb7645e39aba","Type":"ContainerStarted","Data":"019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd"} Nov 03 00:32:34.972419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.972389 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs"] Nov 03 00:32:34.978366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:34.978343 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bb5643-77b879bc79-6btrs"] Nov 03 00:32:35.078074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.077994 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:32:35.079009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.078974 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 00:32:35.079163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.079041 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 00:32:35.255058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.255027 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn"] Nov 03 00:32:35.258957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.258938 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bb5643-6987c869fc-rkgzn"] Nov 03 00:32:35.998307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.998272 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="678b16e7-6c48-4246-81ae-134a663e6046" path="/var/lib/kubelet/pods/678b16e7-6c48-4246-81ae-134a663e6046/volumes" Nov 03 00:32:35.998820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:35.998804 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbd23a01-4668-4afb-8116-f35c52087662" path="/var/lib/kubelet/pods/cbd23a01-4668-4afb-8116-f35c52087662/volumes" Nov 03 00:32:36.077693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:36.077657 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 00:32:36.077914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:36.077716 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 00:32:37.078085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:37.078039 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 00:32:37.078572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:37.078109 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 00:32:38.078171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:38.078138 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 00:32:38.078604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:38.078218 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 00:32:38.553335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:38.553301 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:32:39.078134 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:39.078101 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 00:32:39.078334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:39.078186 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 00:32:40.082410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:40.082379 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:32:40.083810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:40.083783 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:32:42.158257 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:32:42.158220 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:32:47.850760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.850728 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7306-c08719"] Nov 03 00:32:47.851395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851376 5144 cpu_manager.go:414] "containerMap: removing container" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" Nov 03 00:32:47.851438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851400 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" Nov 03 00:32:47.851438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851424 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cbd23a01-4668-4afb-8116-f35c52087662" containerName="payload-pre-processing" Nov 03 00:32:47.851438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851430 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbd23a01-4668-4afb-8116-f35c52087662" containerName="payload-pre-processing" Nov 03 00:32:47.851534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851464 5144 cpu_manager.go:414] "containerMap: removing container" podUID="678b16e7-6c48-4246-81ae-134a663e6046" containerName="payload-processing" Nov 03 00:32:47.851534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851469 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678b16e7-6c48-4246-81ae-134a663e6046" containerName="payload-processing" Nov 03 00:32:47.851602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851575 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="363102b0-1d5d-4235-8a0f-c7acca33b428" containerName="istio-proxy" Nov 03 00:32:47.851602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851587 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbd23a01-4668-4afb-8116-f35c52087662" containerName="payload-pre-processing" Nov 03 00:32:47.851705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.851601 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="678b16e7-6c48-4246-81ae-134a663e6046" containerName="payload-processing" Nov 03 00:32:47.855034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.855017 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:47.909009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.908984 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7306-c08719"] Nov 03 00:32:47.954145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.954104 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz96m\"" Nov 03 00:32:47.957955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:47.957923 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkx77\" (UniqueName: \"kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77\") pod \"test-curl-7306-c08719\" (UID: \"59848969-8d98-48e8-a624-c283b2693bd2\") " pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:48.059888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:48.059859 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkx77\" (UniqueName: \"kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77\") pod \"test-curl-7306-c08719\" (UID: \"59848969-8d98-48e8-a624-c283b2693bd2\") " pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:48.128765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:48.124507 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkx77\" (UniqueName: \"kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77\") pod \"test-curl-7306-c08719\" (UID: \"59848969-8d98-48e8-a624-c283b2693bd2\") " pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:48.166870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:48.166836 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:48.547375 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:32:48.547339 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59848969_8d98_48e8_a624_c283b2693bd2.slice/crio-afce1da3d633fbf97c4e1a3c80e7de4e60b5344c8fbeae03db2cd85f0ccc448b WatchSource:0}: Error finding container afce1da3d633fbf97c4e1a3c80e7de4e60b5344c8fbeae03db2cd85f0ccc448b: Status 404 returned error can't find the container with id afce1da3d633fbf97c4e1a3c80e7de4e60b5344c8fbeae03db2cd85f0ccc448b Nov 03 00:32:48.548025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:48.548000 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7306-c08719"] Nov 03 00:32:48.549937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:48.549921 5144 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 00:32:48.985594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:48.985476 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7306-c08719" event={"ID":"59848969-8d98-48e8-a624-c283b2693bd2","Type":"ContainerStarted","Data":"afce1da3d633fbf97c4e1a3c80e7de4e60b5344c8fbeae03db2cd85f0ccc448b"} Nov 03 00:32:53.046618 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:53.046577 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7306-c08719" event={"ID":"59848969-8d98-48e8-a624-c283b2693bd2","Type":"ContainerStarted","Data":"975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de"} Nov 03 00:32:53.160987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:53.160935 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7306-c08719" podStartSLOduration=2.409803131 podStartE2EDuration="6.160920684s" totalImagesPullingTime="3.751117553s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:53.156907842 +0000 UTC m=+1663.302602298" watchObservedRunningTime="2026-11-03 00:32:53.160920684 +0000 UTC m=+1663.306615140" Nov 03 00:32:54.809121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:54.809033 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7306-c08719"] Nov 03 00:32:54.858540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:54.858505 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7306-c08719"] Nov 03 00:32:55.064415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:55.064304 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7306-c08719" podUID="59848969-8d98-48e8-a624-c283b2693bd2" containerName="test-curl-7306-c08719" containerID="cri-o://975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de" gracePeriod=2 Nov 03 00:32:55.118655 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:32:55.116837 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7306-c08719\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="59848969-8d98-48e8-a624-c283b2693bd2" pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:57.353473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:57.353442 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:57.439386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:57.439353 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77\" (UniqueName: \"kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77\") pod \"59848969-8d98-48e8-a624-c283b2693bd2\" (UID: \"59848969-8d98-48e8-a624-c283b2693bd2\") " Nov 03 00:32:57.441759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:57.441731 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77" pod "59848969-8d98-48e8-a624-c283b2693bd2" (UID: "59848969-8d98-48e8-a624-c283b2693bd2"). InnerVolumeSpecName "kube-api-access-mkx77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:32:57.540971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:57.540930 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mkx77\" (UniqueName: \"kubernetes.io/projected/59848969-8d98-48e8-a624-c283b2693bd2-kube-api-access-mkx77\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:32:57.998043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:57.998007 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59848969-8d98-48e8-a624-c283b2693bd2" path="/var/lib/kubelet/pods/59848969-8d98-48e8-a624-c283b2693bd2/volumes" Nov 03 00:32:58.093584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.093552 5144 generic.go:356] "Generic (PLEG): container finished" podID="59848969-8d98-48e8-a624-c283b2693bd2" containerID="975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de" exitCode=137 Nov 03 00:32:58.093801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.093591 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7306-c08719" Nov 03 00:32:58.093801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.093655 5144 scope.go:122] "RemoveContainer" containerID="975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de" Nov 03 00:32:58.107456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.107437 5144 scope.go:122] "RemoveContainer" containerID="975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de" Nov 03 00:32:58.108709 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:32:58.108679 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de\": container with ID starting with 975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de not found: ID does not exist" containerID="975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de" Nov 03 00:32:58.108807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.108717 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de"} err="failed to get container status \"975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de\": rpc error: code = NotFound desc = could not find container \"975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de\": container with ID starting with 975d6e5f2ca8036e8122dfa9192c73ef31f2ce42f93040e7b38828468c2a28de not found: ID does not exist" Nov 03 00:32:58.640802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.640761 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:32:58.641199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:32:58.640823 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:33:00.123840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:00.123804 5144 generic.go:356] "Generic (PLEG): container finished" podID="3f02589b-9510-4e1e-9535-42cda380ff19" containerID="930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2" exitCode=0 Nov 03 00:33:00.123840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:00.123834 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" event={"ID":"3f02589b-9510-4e1e-9535-42cda380ff19","Type":"ContainerDied","Data":"930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2"} Nov 03 00:33:00.124406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:00.124310 5144 scope.go:122] "RemoveContainer" containerID="930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2" Nov 03 00:33:00.126116 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:00.126083 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:33:00.126215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:00.126166 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:33:00.126215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:00.126210 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:33:01.136092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:01.136049 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" event={"ID":"3f02589b-9510-4e1e-9535-42cda380ff19","Type":"ContainerStarted","Data":"031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a"} Nov 03 00:33:01.542078 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:01.542042 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:33:01.542498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:01.542410 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 00:33:01.542721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:01.542700 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 00:33:02.545038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:02.544998 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 00:33:02.545494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:02.545063 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 00:33:03.541070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:03.541034 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 00:33:03.541394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:03.541366 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 00:33:04.272122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.272088 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h"] Nov 03 00:33:04.272793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.272778 5144 cpu_manager.go:414] "containerMap: removing container" podUID="59848969-8d98-48e8-a624-c283b2693bd2" containerName="test-curl-7306-c08719" Nov 03 00:33:04.272871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.272794 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59848969-8d98-48e8-a624-c283b2693bd2" containerName="test-curl-7306-c08719" Nov 03 00:33:04.272943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.272932 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="59848969-8d98-48e8-a624-c283b2693bd2" containerName="test-curl-7306-c08719" Nov 03 00:33:04.277587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.277560 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.334424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.330427 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1a883c-openshift-default-dockercfg-zwtcb\"" Nov 03 00:33:04.388659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.367210 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.446752 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8g2\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.446821 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.446880 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.447010 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.447075 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.447128 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.447170 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.447657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.447240 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.580060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.579814 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 00:33:04.580378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.580349 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.580654 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.580705 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.580769 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.580913 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.580962 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8g2\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.581000 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.581053 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.581155 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.581660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.581214 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.582225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.582091 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.582528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.582507 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.582703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.582609 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h"] Nov 03 00:33:04.582927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.582904 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.583184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.583160 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.583716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.583697 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.592707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.592676 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.592946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.592922 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.646400 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.646357 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8g2\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.647067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.647043 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token\") pod \"e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:04.653503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:04.653480 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:05.298822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:05.298789 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h"] Nov 03 00:33:05.355720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:05.355684 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:33:05.355844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:05.355764 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:33:05.355844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:05.355794 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:33:05.540668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:05.540038 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 00:33:05.540668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:05.540099 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 00:33:06.204658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.204598 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" event={"ID":"1d75a7a8-7808-4e76-94b1-be98213ea910","Type":"ContainerStarted","Data":"eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3"} Nov 03 00:33:06.204836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.204682 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" event={"ID":"1d75a7a8-7808-4e76-94b1-be98213ea910","Type":"ContainerStarted","Data":"d3e772108455c1339c89923e657605f847e54bb5177065023a89ccb744dba140"} Nov 03 00:33:06.540050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.540009 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" start-of-body= Nov 03 00:33:06.540662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.540606 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.167:15021/healthz/ready\": dial tcp 10.132.0.167:15021: connect: connection refused" Nov 03 00:33:06.653906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.653870 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:06.659358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.659324 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:06.779357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:06.779296 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podStartSLOduration=2.779276808 podStartE2EDuration="2.779276808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:06.322934218 +0000 UTC m=+1676.468628676" watchObservedRunningTime="2026-11-03 00:33:06.779276808 +0000 UTC m=+1676.924971267" Nov 03 00:33:07.216371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:07.215085 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:07.216750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:07.216719 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:33:07.543533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:07.543504 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:33:07.544003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:07.543980 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:33:07.544648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:07.544612 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:33:09.794277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:09.794244 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb"] Nov 03 00:33:09.905060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:09.905027 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q"] Nov 03 00:33:12.158833 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:12.158792 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:33:14.795125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:14.795080 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" podUID="95d709e4-4ce5-4fda-ada1-92775f3315e1" containerName="payload-processing" containerID="cri-o://5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f" gracePeriod=25 Nov 03 00:33:14.906166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:14.906134 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" podUID="56252b9f-c05f-4a2b-b615-18587c611828" containerName="payload-pre-processing" containerID="cri-o://b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6" gracePeriod=25 Nov 03 00:33:15.089430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.089399 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:33:15.197476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.197379 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw\" (UniqueName: \"kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw\") pod \"95d709e4-4ce5-4fda-ada1-92775f3315e1\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " Nov 03 00:33:15.197598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.197528 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume\") pod \"95d709e4-4ce5-4fda-ada1-92775f3315e1\" (UID: \"95d709e4-4ce5-4fda-ada1-92775f3315e1\") " Nov 03 00:33:15.197916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.197892 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume" pod "95d709e4-4ce5-4fda-ada1-92775f3315e1" (UID: "95d709e4-4ce5-4fda-ada1-92775f3315e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:15.198082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.198067 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/95d709e4-4ce5-4fda-ada1-92775f3315e1-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.199779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.199755 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw" pod "95d709e4-4ce5-4fda-ada1-92775f3315e1" (UID: "95d709e4-4ce5-4fda-ada1-92775f3315e1"). InnerVolumeSpecName "kube-api-access-wcpxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:15.211085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.211061 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:33:15.300712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.300285 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s\" (UniqueName: \"kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s\") pod \"56252b9f-c05f-4a2b-b615-18587c611828\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " Nov 03 00:33:15.300712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.300453 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume\") pod \"56252b9f-c05f-4a2b-b615-18587c611828\" (UID: \"56252b9f-c05f-4a2b-b615-18587c611828\") " Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.301138 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcpxw\" (UniqueName: \"kubernetes.io/projected/95d709e4-4ce5-4fda-ada1-92775f3315e1-kube-api-access-wcpxw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.301670 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume" pod "56252b9f-c05f-4a2b-b615-18587c611828" (UID: "56252b9f-c05f-4a2b-b615-18587c611828"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.309271 5144 generic.go:356] "Generic (PLEG): container finished" podID="95d709e4-4ce5-4fda-ada1-92775f3315e1" containerID="5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f" exitCode=0 Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.309515 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" event={"ID":"95d709e4-4ce5-4fda-ada1-92775f3315e1","Type":"ContainerDied","Data":"5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f"} Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.309564 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" event={"ID":"95d709e4-4ce5-4fda-ada1-92775f3315e1","Type":"ContainerDied","Data":"3b3111582a5cc4bfccf56fcc08bdbb36dab75b9cc99bf4b258b31b25ffabed70"} Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.309591 5144 scope.go:122] "RemoveContainer" containerID="5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f" Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.309778 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb" Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.315790 5144 generic.go:356] "Generic (PLEG): container finished" podID="56252b9f-c05f-4a2b-b615-18587c611828" containerID="b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6" exitCode=0 Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.316071 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" event={"ID":"56252b9f-c05f-4a2b-b615-18587c611828","Type":"ContainerDied","Data":"b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6"} Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.316207 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" event={"ID":"56252b9f-c05f-4a2b-b615-18587c611828","Type":"ContainerDied","Data":"c0b243498d6fd82455e021a1216d22e5ea353ae9bd2e0257aaf2bda20358d03f"} Nov 03 00:33:15.330658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.316374 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q" Nov 03 00:33:15.362821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.361498 5144 scope.go:122] "RemoveContainer" containerID="5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f" Nov 03 00:33:15.362821 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:15.362102 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f\": container with ID starting with 5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f not found: ID does not exist" containerID="5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f" Nov 03 00:33:15.362821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.362134 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f"} err="failed to get container status \"5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f\": rpc error: code = NotFound desc = could not find container \"5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f\": container with ID starting with 5a661a51174c5658219e689a395dcff18c7098df11da13d5c3f3adc4e76b9d9f not found: ID does not exist" Nov 03 00:33:15.362821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.362157 5144 scope.go:122] "RemoveContainer" containerID="b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6" Nov 03 00:33:15.386655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.365136 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s" pod "56252b9f-c05f-4a2b-b615-18587c611828" (UID: "56252b9f-c05f-4a2b-b615-18587c611828"). InnerVolumeSpecName "kube-api-access-fjw2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:15.404934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.404256 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjw2s\" (UniqueName: \"kubernetes.io/projected/56252b9f-c05f-4a2b-b615-18587c611828-kube-api-access-fjw2s\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.404934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.404298 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56252b9f-c05f-4a2b-b615-18587c611828-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:15.415968 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.415945 5144 scope.go:122] "RemoveContainer" containerID="b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6" Nov 03 00:33:15.416377 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:15.416347 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6\": container with ID starting with b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6 not found: ID does not exist" containerID="b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6" Nov 03 00:33:15.416468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.416390 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6"} err="failed to get container status \"b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6\": rpc error: code = NotFound desc = could not find container \"b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6\": container with ID starting with b4d34bc6aa0a9c0a8548399aad59269740430ffaadd0a2f14dbe033406ed68c6 not found: ID does not exist" Nov 03 00:33:15.591220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.591125 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb"] Nov 03 00:33:15.597048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.597022 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-6bbc6c49c5-nmdvb"] Nov 03 00:33:15.742061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.742032 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q"] Nov 03 00:33:15.748261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.748237 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-8c9581-5cc94dd49f-xg84q"] Nov 03 00:33:15.996553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.996476 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56252b9f-c05f-4a2b-b615-18587c611828" path="/var/lib/kubelet/pods/56252b9f-c05f-4a2b-b615-18587c611828/volumes" Nov 03 00:33:15.997039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:15.997019 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95d709e4-4ce5-4fda-ada1-92775f3315e1" path="/var/lib/kubelet/pods/95d709e4-4ce5-4fda-ada1-92775f3315e1/volumes" Nov 03 00:33:17.659469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:17.658583 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn"] Nov 03 00:33:22.659123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:22.659063 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" podUID="6905a5fa-c3e0-4904-8021-23503d9dc816" containerName="payload-pre-processing" containerID="cri-o://efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768" gracePeriod=25 Nov 03 00:33:22.954602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:22.954572 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:33:23.117467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.117427 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2\" (UniqueName: \"kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2\") pod \"6905a5fa-c3e0-4904-8021-23503d9dc816\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " Nov 03 00:33:23.117669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.117512 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume\") pod \"6905a5fa-c3e0-4904-8021-23503d9dc816\" (UID: \"6905a5fa-c3e0-4904-8021-23503d9dc816\") " Nov 03 00:33:23.117918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.117890 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume" pod "6905a5fa-c3e0-4904-8021-23503d9dc816" (UID: "6905a5fa-c3e0-4904-8021-23503d9dc816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:23.118249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.118228 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6905a5fa-c3e0-4904-8021-23503d9dc816-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:23.119948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.119926 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2" pod "6905a5fa-c3e0-4904-8021-23503d9dc816" (UID: "6905a5fa-c3e0-4904-8021-23503d9dc816"). InnerVolumeSpecName "kube-api-access-9stt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:23.219570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.219510 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9stt2\" (UniqueName: \"kubernetes.io/projected/6905a5fa-c3e0-4904-8021-23503d9dc816-kube-api-access-9stt2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:23.425447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.425411 5144 generic.go:356] "Generic (PLEG): container finished" podID="6905a5fa-c3e0-4904-8021-23503d9dc816" containerID="efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768" exitCode=0 Nov 03 00:33:23.425647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.425459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" Nov 03 00:33:23.425647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.425503 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" event={"ID":"6905a5fa-c3e0-4904-8021-23503d9dc816","Type":"ContainerDied","Data":"efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768"} Nov 03 00:33:23.425647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.425555 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn" event={"ID":"6905a5fa-c3e0-4904-8021-23503d9dc816","Type":"ContainerDied","Data":"79f1bff355fd091e9d80328d4adb39b91d02d115afe29a1ba1e528aa8a057947"} Nov 03 00:33:23.425647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.425575 5144 scope.go:122] "RemoveContainer" containerID="efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768" Nov 03 00:33:23.438297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.438280 5144 scope.go:122] "RemoveContainer" containerID="efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768" Nov 03 00:33:23.438585 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:23.438566 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768\": container with ID starting with efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768 not found: ID does not exist" containerID="efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768" Nov 03 00:33:23.438683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.438595 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768"} err="failed to get container status \"efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768\": rpc error: code = NotFound desc = could not find container \"efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768\": container with ID starting with efaaa1474f8dca6c1377abbccece77211a18bb30f344d77d35eeed4a6b544768 not found: ID does not exist" Nov 03 00:33:23.593664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.593624 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn"] Nov 03 00:33:23.643217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.643184 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-3df441-65d6685d99-x2vzn"] Nov 03 00:33:23.997265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:23.997186 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6905a5fa-c3e0-4904-8021-23503d9dc816" path="/var/lib/kubelet/pods/6905a5fa-c3e0-4904-8021-23503d9dc816/volumes" Nov 03 00:33:33.049738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:33.049704 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl"] Nov 03 00:33:33.096554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:33.096525 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f5ac2d31-cj7gl"] Nov 03 00:33:33.996917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:33.996885 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe" path="/var/lib/kubelet/pods/15b3b8cd-8aea-4c21-a1e9-5ba4c9b13bfe/volumes" Nov 03 00:33:38.004850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:38.004819 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g"] Nov 03 00:33:38.111138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:38.111108 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw"] Nov 03 00:33:42.158608 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:42.158562 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:33:43.006110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.006045 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" podUID="5ba65ac5-5c99-4256-b9ba-05a111030b88" containerName="payload-processing" containerID="cri-o://294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617" gracePeriod=25 Nov 03 00:33:43.111774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.111740 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" podUID="7c09bab9-c0d9-4845-9221-443c89cc7005" containerName="payload-pre-processing" containerID="cri-o://85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607" gracePeriod=25 Nov 03 00:33:43.304611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.304587 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:33:43.368822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.368787 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx\" (UniqueName: \"kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx\") pod \"5ba65ac5-5c99-4256-b9ba-05a111030b88\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " Nov 03 00:33:43.369001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.368970 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume\") pod \"5ba65ac5-5c99-4256-b9ba-05a111030b88\" (UID: \"5ba65ac5-5c99-4256-b9ba-05a111030b88\") " Nov 03 00:33:43.369427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.369398 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume" pod "5ba65ac5-5c99-4256-b9ba-05a111030b88" (UID: "5ba65ac5-5c99-4256-b9ba-05a111030b88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:43.369670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.369622 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ba65ac5-5c99-4256-b9ba-05a111030b88-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:43.372812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.372784 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx" pod "5ba65ac5-5c99-4256-b9ba-05a111030b88" (UID: "5ba65ac5-5c99-4256-b9ba-05a111030b88"). InnerVolumeSpecName "kube-api-access-94mgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:43.401115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.401092 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:33:43.470109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.470075 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs\" (UniqueName: \"kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs\") pod \"7c09bab9-c0d9-4845-9221-443c89cc7005\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " Nov 03 00:33:43.470343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.470318 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume\") pod \"7c09bab9-c0d9-4845-9221-443c89cc7005\" (UID: \"7c09bab9-c0d9-4845-9221-443c89cc7005\") " Nov 03 00:33:43.470722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.470696 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume" pod "7c09bab9-c0d9-4845-9221-443c89cc7005" (UID: "7c09bab9-c0d9-4845-9221-443c89cc7005"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:43.470875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.470860 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94mgx\" (UniqueName: \"kubernetes.io/projected/5ba65ac5-5c99-4256-b9ba-05a111030b88-kube-api-access-94mgx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:43.470920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.470877 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c09bab9-c0d9-4845-9221-443c89cc7005-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:43.472276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.472254 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs" pod "7c09bab9-c0d9-4845-9221-443c89cc7005" (UID: "7c09bab9-c0d9-4845-9221-443c89cc7005"). InnerVolumeSpecName "kube-api-access-qpfvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:43.572544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.572482 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qpfvs\" (UniqueName: \"kubernetes.io/projected/7c09bab9-c0d9-4845-9221-443c89cc7005-kube-api-access-qpfvs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:43.593204 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.593175 5144 generic.go:356] "Generic (PLEG): container finished" podID="5ba65ac5-5c99-4256-b9ba-05a111030b88" containerID="294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617" exitCode=0 Nov 03 00:33:43.593339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.593217 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" Nov 03 00:33:43.593339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.593265 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" event={"ID":"5ba65ac5-5c99-4256-b9ba-05a111030b88","Type":"ContainerDied","Data":"294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617"} Nov 03 00:33:43.593339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.593326 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g" event={"ID":"5ba65ac5-5c99-4256-b9ba-05a111030b88","Type":"ContainerDied","Data":"b04f7f874d67ac701505f3a75550d1e180b71f0af099cc6650800da2c4b3e2a1"} Nov 03 00:33:43.593517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.593355 5144 scope.go:122] "RemoveContainer" containerID="294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617" Nov 03 00:33:43.595131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.595055 5144 generic.go:356] "Generic (PLEG): container finished" podID="7c09bab9-c0d9-4845-9221-443c89cc7005" containerID="85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607" exitCode=0 Nov 03 00:33:43.595131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.595094 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" event={"ID":"7c09bab9-c0d9-4845-9221-443c89cc7005","Type":"ContainerDied","Data":"85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607"} Nov 03 00:33:43.595131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.595098 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" Nov 03 00:33:43.595131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.595131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw" event={"ID":"7c09bab9-c0d9-4845-9221-443c89cc7005","Type":"ContainerDied","Data":"5eb0e7265afce6d0358df3f54db0f876d97ac899fefa835dd0aa36a013e73c49"} Nov 03 00:33:43.604115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.604096 5144 scope.go:122] "RemoveContainer" containerID="294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617" Nov 03 00:33:43.604360 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:43.604341 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617\": container with ID starting with 294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617 not found: ID does not exist" containerID="294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617" Nov 03 00:33:43.604416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.604370 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617"} err="failed to get container status \"294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617\": rpc error: code = NotFound desc = could not find container \"294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617\": container with ID starting with 294818c77e161e07666a28b4e2d5a04166698a4c8d1101c91ef1514f9b38b617 not found: ID does not exist" Nov 03 00:33:43.604416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.604386 5144 scope.go:122] "RemoveContainer" containerID="85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607" Nov 03 00:33:43.613658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.613611 5144 scope.go:122] "RemoveContainer" containerID="85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607" Nov 03 00:33:43.613909 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:33:43.613891 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607\": container with ID starting with 85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607 not found: ID does not exist" containerID="85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607" Nov 03 00:33:43.613976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.613915 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607"} err="failed to get container status \"85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607\": rpc error: code = NotFound desc = could not find container \"85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607\": container with ID starting with 85e41f8e8b518c7e65a0919521ae7201bdacacd3ccdf8d6082502a4af59ea607 not found: ID does not exist" Nov 03 00:33:43.759959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.759917 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw"] Nov 03 00:33:43.765821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.765798 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-cb65f3a5-5899694f4c-klffw"] Nov 03 00:33:43.873905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.873835 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g"] Nov 03 00:33:43.879055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.879032 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-cb65f3a5-66fb4bd849-8r49g"] Nov 03 00:33:43.997102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.997074 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba65ac5-5c99-4256-b9ba-05a111030b88" path="/var/lib/kubelet/pods/5ba65ac5-5c99-4256-b9ba-05a111030b88/volumes" Nov 03 00:33:43.997457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:43.997444 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c09bab9-c0d9-4845-9221-443c89cc7005" path="/var/lib/kubelet/pods/7c09bab9-c0d9-4845-9221-443c89cc7005/volumes" Nov 03 00:33:46.794357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.794317 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np"] Nov 03 00:33:46.795072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795050 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6905a5fa-c3e0-4904-8021-23503d9dc816" containerName="payload-pre-processing" Nov 03 00:33:46.795072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795069 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6905a5fa-c3e0-4904-8021-23503d9dc816" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795089 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5ba65ac5-5c99-4256-b9ba-05a111030b88" containerName="payload-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795097 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba65ac5-5c99-4256-b9ba-05a111030b88" containerName="payload-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795107 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7c09bab9-c0d9-4845-9221-443c89cc7005" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795113 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c09bab9-c0d9-4845-9221-443c89cc7005" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795149 5144 cpu_manager.go:414] "containerMap: removing container" podUID="95d709e4-4ce5-4fda-ada1-92775f3315e1" containerName="payload-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795154 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95d709e4-4ce5-4fda-ada1-92775f3315e1" containerName="payload-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795167 5144 cpu_manager.go:414] "containerMap: removing container" podUID="56252b9f-c05f-4a2b-b615-18587c611828" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795173 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56252b9f-c05f-4a2b-b615-18587c611828" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795283 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba65ac5-5c99-4256-b9ba-05a111030b88" containerName="payload-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795292 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c09bab9-c0d9-4845-9221-443c89cc7005" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795303 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="95d709e4-4ce5-4fda-ada1-92775f3315e1" containerName="payload-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795309 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="56252b9f-c05f-4a2b-b615-18587c611828" containerName="payload-pre-processing" Nov 03 00:33:46.795419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.795318 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6905a5fa-c3e0-4904-8021-23503d9dc816" containerName="payload-pre-processing" Nov 03 00:33:46.800338 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.800316 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:46.850604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.850573 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np"] Nov 03 00:33:46.895722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.895698 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-eb20df\"" Nov 03 00:33:46.895890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.895783 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-eb20df-dockercfg-n8rg5\"" Nov 03 00:33:46.919935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.919902 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7"] Nov 03 00:33:46.921997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.921962 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n45kl\" (UniqueName: \"kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl\") pod \"payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:46.922185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.922167 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:46.924321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.924297 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.032876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:46.989067 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7"] Nov 03 00:33:47.032876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.028498 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n45kl\" (UniqueName: \"kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl\") pod \"payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:47.032876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.028563 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.032876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.028660 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44kkh\" (UniqueName: \"kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh\") pod \"payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.032876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.028737 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:47.032876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.029789 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:47.088745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.088616 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n45kl\" (UniqueName: \"kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl\") pod \"payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:47.118781 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.118750 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:47.136613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.136576 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.139056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.139031 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44kkh\" (UniqueName: \"kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh\") pod \"payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.143111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.140607 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.208686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.207610 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44kkh\" (UniqueName: \"kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh\") pod \"payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.303460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.303222 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:47.583721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.583696 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np"] Nov 03 00:33:47.585286 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:33:47.585258 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9187ddba_9bec_4b7d_b081_482223b24fcf.slice/crio-a8c331fc65710d9f266dfca1bf058d3717dc6ed475218cba82ba4f0830888f25 WatchSource:0}: Error finding container a8c331fc65710d9f266dfca1bf058d3717dc6ed475218cba82ba4f0830888f25: Status 404 returned error can't find the container with id a8c331fc65710d9f266dfca1bf058d3717dc6ed475218cba82ba4f0830888f25 Nov 03 00:33:47.634402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.634372 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" event={"ID":"9187ddba-9bec-4b7d-b081-482223b24fcf","Type":"ContainerStarted","Data":"a8c331fc65710d9f266dfca1bf058d3717dc6ed475218cba82ba4f0830888f25"} Nov 03 00:33:47.697008 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:33:47.696983 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e09516_4b77_4e8e_8174_8c27a69aa96d.slice/crio-d6acb4401ae758ab21dda1649f7cc5567246fec7889202bbd7dabebe055647ce WatchSource:0}: Error finding container d6acb4401ae758ab21dda1649f7cc5567246fec7889202bbd7dabebe055647ce: Status 404 returned error can't find the container with id d6acb4401ae758ab21dda1649f7cc5567246fec7889202bbd7dabebe055647ce Nov 03 00:33:47.697112 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:47.697092 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7"] Nov 03 00:33:48.649492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.649452 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" event={"ID":"a8e09516-4b77-4e8e-8174-8c27a69aa96d","Type":"ContainerStarted","Data":"50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811"} Nov 03 00:33:48.650019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.649508 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" event={"ID":"a8e09516-4b77-4e8e-8174-8c27a69aa96d","Type":"ContainerStarted","Data":"d6acb4401ae758ab21dda1649f7cc5567246fec7889202bbd7dabebe055647ce"} Nov 03 00:33:48.650019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.649557 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:48.651483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.651451 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" event={"ID":"9187ddba-9bec-4b7d-b081-482223b24fcf","Type":"ContainerStarted","Data":"5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0"} Nov 03 00:33:48.651652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.651538 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:33:48.765791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.765738 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" podStartSLOduration=2.765719406 podStartE2EDuration="2.765719406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:48.76403776 +0000 UTC m=+1718.909732233" watchObservedRunningTime="2026-11-03 00:33:48.765719406 +0000 UTC m=+1718.911413862" Nov 03 00:33:48.880864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:48.880811 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" podStartSLOduration=2.880797574 podStartE2EDuration="2.880797574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:48.877106709 +0000 UTC m=+1719.022801203" watchObservedRunningTime="2026-11-03 00:33:48.880797574 +0000 UTC m=+1719.026492030" Nov 03 00:33:50.549225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:50.549198 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72"] Nov 03 00:33:50.549678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:50.549383 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" containerID="cri-o://5083af0e35495acaedb304e31142812baa9086ba7a62418c2eef54ce64bc48a0" gracePeriod=30 Nov 03 00:33:54.491972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:54.491929 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:33:54.492366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:54.492000 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:33:55.784809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.784772 5144 generic.go:356] "Generic (PLEG): container finished" podID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerID="5083af0e35495acaedb304e31142812baa9086ba7a62418c2eef54ce64bc48a0" exitCode=0 Nov 03 00:33:55.785287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.784833 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" event={"ID":"e6f913c4-cfcf-4f8a-96ef-66f0debc631f","Type":"ContainerDied","Data":"5083af0e35495acaedb304e31142812baa9086ba7a62418c2eef54ce64bc48a0"} Nov 03 00:33:55.785287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.784891 5144 scope.go:122] "RemoveContainer" containerID="ae33c85f3e629291fdedc712152ee66db826d970f60cfab53a48746b159dc915" Nov 03 00:33:55.834667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.834617 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:33:55.968209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968173 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968222 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968300 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968350 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968369 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968442 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968593 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968621 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968690 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token\") pod \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\" (UID: \"e6f913c4-cfcf-4f8a-96ef-66f0debc631f\") " Nov 03 00:33:55.968813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968726 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:33:55.968951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.968905 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:55.969394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.969221 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:55.969522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.969487 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:55.969522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.969508 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:55.969522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.969521 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:55.970096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.969587 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:55.970288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.970238 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:55.971043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.971012 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:33:55.971553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.971529 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "kube-api-access-5pg47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:55.971812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.971788 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:33:55.971812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:55.971790 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token" pod "e6f913c4-cfcf-4f8a-96ef-66f0debc631f" (UID: "e6f913c4-cfcf-4f8a-96ef-66f0debc631f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:33:56.070610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.070530 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:56.070610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.070567 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pg47\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-kube-api-access-5pg47\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:56.070610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.070586 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:56.070610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.070601 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:56.070937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.070614 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:56.070937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.070664 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6f913c4-cfcf-4f8a-96ef-66f0debc631f-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:33:56.793956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.793924 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" event={"ID":"e6f913c4-cfcf-4f8a-96ef-66f0debc631f","Type":"ContainerDied","Data":"6fc76c3f631e6d48d94505adf7303beb3ca028848edc2eef1e4cdc186afeae14"} Nov 03 00:33:56.794405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.793968 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72" Nov 03 00:33:56.794405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:56.793981 5144 scope.go:122] "RemoveContainer" containerID="5083af0e35495acaedb304e31142812baa9086ba7a62418c2eef54ce64bc48a0" Nov 03 00:33:57.007447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:57.007405 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72"] Nov 03 00:33:57.057236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:57.057161 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-38f5a3-openshift-default-57d476fb47-q9f72"] Nov 03 00:33:57.997422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:57.997392 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" path="/var/lib/kubelet/pods/e6f913c4-cfcf-4f8a-96ef-66f0debc631f/volumes" Nov 03 00:33:59.666189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:59.666165 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:33:59.666583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:33:59.666235 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:34:00.396222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.396190 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8"] Nov 03 00:34:00.396838 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.396824 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" Nov 03 00:34:00.396894 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.396839 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" Nov 03 00:34:00.396973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.396964 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" Nov 03 00:34:00.400409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.400395 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.456331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.456310 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8"] Nov 03 00:34:00.501046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.501022 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-8c9581\"" Nov 03 00:34:00.501046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.501035 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-8c9581-dockercfg-r98b2\"" Nov 03 00:34:00.541061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.541031 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.541217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.541074 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj5ff\" (UniqueName: \"kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff\") pod \"payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.642922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.642895 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.643058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.642929 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj5ff\" (UniqueName: \"kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff\") pod \"payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.643776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.643754 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.699924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.699863 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj5ff\" (UniqueName: \"kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff\") pod \"payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:00.714250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:00.714226 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:01.102385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:01.102357 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8"] Nov 03 00:34:01.102845 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:34:01.102817 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b55ffcb_0693_42f3_a605_142c3d026224.slice/crio-ef04350d18b4ffbc529b9436c4235499d22b1d16751bfe69a4068069554db6cf WatchSource:0}: Error finding container ef04350d18b4ffbc529b9436c4235499d22b1d16751bfe69a4068069554db6cf: Status 404 returned error can't find the container with id ef04350d18b4ffbc529b9436c4235499d22b1d16751bfe69a4068069554db6cf Nov 03 00:34:01.860707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:01.860674 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" event={"ID":"3b55ffcb-0693-42f3-a605-142c3d026224","Type":"ContainerStarted","Data":"ef04350d18b4ffbc529b9436c4235499d22b1d16751bfe69a4068069554db6cf"} Nov 03 00:34:03.882673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:03.882620 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" event={"ID":"3b55ffcb-0693-42f3-a605-142c3d026224","Type":"ContainerStarted","Data":"79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3"} Nov 03 00:34:03.883108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:03.882747 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:03.995951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:03.995807 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" podStartSLOduration=1.791071168 podStartE2EDuration="3.995790422s" totalImagesPullingTime="2.204719254s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:03.995082272 +0000 UTC m=+1734.140776729" watchObservedRunningTime="2026-11-03 00:34:03.995790422 +0000 UTC m=+1734.141484877" Nov 03 00:34:12.157695 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:34:12.157661 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:34:14.892490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:14.892462 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:25.866141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.866100 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg"] Nov 03 00:34:25.867735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.867711 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" Nov 03 00:34:25.867890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.867878 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" Nov 03 00:34:25.868165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.868151 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6f913c4-cfcf-4f8a-96ef-66f0debc631f" containerName="istio-proxy" Nov 03 00:34:25.872413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.872396 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:25.899927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.899862 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:25.900070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.900035 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pv6f\" (UniqueName: \"kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:25.900168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.900151 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:25.971371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.971332 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg"] Nov 03 00:34:25.975650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.975539 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:34:25.975650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:25.975608 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:26.001439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.001401 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.001578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.001510 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.001624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.001590 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pv6f\" (UniqueName: \"kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.002384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.002277 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.006195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.006170 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.059495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.059464 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pv6f\" (UniqueName: \"kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f\") pod \"maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.194907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.194830 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:26.596922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:26.596891 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg"] Nov 03 00:34:26.597428 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:34:26.597400 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5361277_f4f4_4267_85ca_2f2a6256ab6c.slice/crio-71f8a36bd4c5ace5ff789e144a2c4979b47d3d7ed8db9edf017ab42a243a49a3 WatchSource:0}: Error finding container 71f8a36bd4c5ace5ff789e144a2c4979b47d3d7ed8db9edf017ab42a243a49a3: Status 404 returned error can't find the container with id 71f8a36bd4c5ace5ff789e144a2c4979b47d3d7ed8db9edf017ab42a243a49a3 Nov 03 00:34:27.101550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:27.101518 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" event={"ID":"a5361277-f4f4-4267-85ca-2f2a6256ab6c","Type":"ContainerStarted","Data":"71f8a36bd4c5ace5ff789e144a2c4979b47d3d7ed8db9edf017ab42a243a49a3"} Nov 03 00:34:28.112467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:28.112381 5144 generic.go:356] "Generic (PLEG): container finished" podID="a5361277-f4f4-4267-85ca-2f2a6256ab6c" containerID="80801308821cd1ade113ed372f3b6ab6cf75a7576d391cf14f4403bd80ca7e6c" exitCode=0 Nov 03 00:34:28.112833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:28.112469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" event={"ID":"a5361277-f4f4-4267-85ca-2f2a6256ab6c","Type":"ContainerDied","Data":"80801308821cd1ade113ed372f3b6ab6cf75a7576d391cf14f4403bd80ca7e6c"} Nov 03 00:34:29.293433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.293408 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:29.375839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.375762 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca\") pod \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " Nov 03 00:34:29.375992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.375868 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca\") pod \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " Nov 03 00:34:29.375992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.375897 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f\" (UniqueName: \"kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f\") pod \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\" (UID: \"a5361277-f4f4-4267-85ca-2f2a6256ab6c\") " Nov 03 00:34:29.376591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.376223 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca" pod "a5361277-f4f4-4267-85ca-2f2a6256ab6c" (UID: "a5361277-f4f4-4267-85ca-2f2a6256ab6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:29.376591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.376565 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:29.378064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.378037 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca" pod "a5361277-f4f4-4267-85ca-2f2a6256ab6c" (UID: "a5361277-f4f4-4267-85ca-2f2a6256ab6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:29.378188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.378158 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f" pod "a5361277-f4f4-4267-85ca-2f2a6256ab6c" (UID: "a5361277-f4f4-4267-85ca-2f2a6256ab6c"). InnerVolumeSpecName "kube-api-access-9pv6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:29.478469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.478439 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pv6f\" (UniqueName: \"kubernetes.io/projected/a5361277-f4f4-4267-85ca-2f2a6256ab6c-kube-api-access-9pv6f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:29.478469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:29.478468 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a5361277-f4f4-4267-85ca-2f2a6256ab6c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:30.129813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:30.129773 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" Nov 03 00:34:30.129813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:30.129798 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg" event={"ID":"a5361277-f4f4-4267-85ca-2f2a6256ab6c","Type":"ContainerDied","Data":"71f8a36bd4c5ace5ff789e144a2c4979b47d3d7ed8db9edf017ab42a243a49a3"} Nov 03 00:34:30.130024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:30.129854 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71f8a36bd4c5ace5ff789e144a2c4979b47d3d7ed8db9edf017ab42a243a49a3" Nov 03 00:34:33.158885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:33.158852 5144 generic.go:356] "Generic (PLEG): container finished" podID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerID="eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3" exitCode=0 Nov 03 00:34:33.159842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:33.158926 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" event={"ID":"1d75a7a8-7808-4e76-94b1-be98213ea910","Type":"ContainerDied","Data":"eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3"} Nov 03 00:34:33.159842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:33.159299 5144 scope.go:122] "RemoveContainer" containerID="eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3" Nov 03 00:34:33.160834 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:33.160806 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:34:33.160899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:33.160876 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:34:33.160939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:33.160908 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:34:34.167577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:34.167541 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" event={"ID":"1d75a7a8-7808-4e76-94b1-be98213ea910","Type":"ContainerStarted","Data":"006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b"} Nov 03 00:34:34.666517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:34.666472 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:34:34.667591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:34.667558 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 00:34:34.667839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:34.667805 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 00:34:35.654171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:35.654128 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 00:34:35.654600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:35.654197 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 00:34:36.153610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.153547 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79"] Nov 03 00:34:36.154316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.154300 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a5361277-f4f4-4267-85ca-2f2a6256ab6c" containerName="revoke-keys" Nov 03 00:34:36.154374 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.154317 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5361277-f4f4-4267-85ca-2f2a6256ab6c" containerName="revoke-keys" Nov 03 00:34:36.154456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.154447 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5361277-f4f4-4267-85ca-2f2a6256ab6c" containerName="revoke-keys" Nov 03 00:34:36.157253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.157222 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.211782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.211745 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212051 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212099 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212196 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212239 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212350 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq587\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212398 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.212715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.212428 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.214260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.214237 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-f2d039-openshift-default-dockercfg-hvr6m\"" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384362 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384440 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384484 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384573 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384646 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384791 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq587\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384845 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384879 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.384950 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.385575 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.385849 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.386092 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.396656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.391286 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.401734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.398973 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79"] Nov 03 00:34:36.401734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.399970 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.412716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.404702 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.412716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.410832 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.455372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.455338 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq587\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.455812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.455794 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token\") pod \"e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.511504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.511478 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:36.660942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.654477 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 00:34:36.660942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:36.654538 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 00:34:37.006981 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:34:37.006954 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81b79a89_52ca_4440_9874_cc9a78bc0735.slice/crio-43d65a91f20069630ce8e267ed00d9e146d9718dab3d2f22509325ad8c3b6628 WatchSource:0}: Error finding container 43d65a91f20069630ce8e267ed00d9e146d9718dab3d2f22509325ad8c3b6628: Status 404 returned error can't find the container with id 43d65a91f20069630ce8e267ed00d9e146d9718dab3d2f22509325ad8c3b6628 Nov 03 00:34:37.007582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.007553 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79"] Nov 03 00:34:37.009484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.009453 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:34:37.009609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.009520 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:34:37.009609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.009564 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:34:37.218561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.217716 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:34:37.263199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.263106 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" event={"ID":"81b79a89-52ca-4440-9874-cc9a78bc0735","Type":"ContainerStarted","Data":"e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017"} Nov 03 00:34:37.263199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.263159 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" event={"ID":"81b79a89-52ca-4440-9874-cc9a78bc0735","Type":"ContainerStarted","Data":"43d65a91f20069630ce8e267ed00d9e146d9718dab3d2f22509325ad8c3b6628"} Nov 03 00:34:37.386700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.386644 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podStartSLOduration=1.3866167759999999 podStartE2EDuration="1.386616776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:37.381794785 +0000 UTC m=+1767.527489240" watchObservedRunningTime="2026-11-03 00:34:37.386616776 +0000 UTC m=+1767.532311232" Nov 03 00:34:37.515714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.514725 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:37.654711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.654678 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 00:34:37.654892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:37.654738 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 00:34:38.540656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:38.539916 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:38.655655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:38.655011 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" start-of-body= Nov 03 00:34:38.655655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:38.655073 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.173:15021/healthz/ready\": dial tcp 10.132.0.173:15021: connect: connection refused" Nov 03 00:34:39.328276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:39.328241 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:39.332276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:39.332243 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:34:39.658445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:39.658372 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:34:39.659661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:39.659609 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:34:40.310612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:40.310569 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg"] Nov 03 00:34:40.314075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:40.314048 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-cb65f3a5-jkjxg"] Nov 03 00:34:40.560242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:40.560200 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b"] Nov 03 00:34:40.560475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:40.560431 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" containerID="cri-o://d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57" gracePeriod=30 Nov 03 00:34:40.567240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:40.567210 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d"] Nov 03 00:34:40.567431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:40.567407 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" containerID="cri-o://af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f" gracePeriod=30 Nov 03 00:34:41.998338 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:41.998305 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5361277-f4f4-4267-85ca-2f2a6256ab6c" path="/var/lib/kubelet/pods/a5361277-f4f4-4267-85ca-2f2a6256ab6c/volumes" Nov 03 00:34:42.156179 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:34:42.156146 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:34:45.110537 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.110509 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x"] Nov 03 00:34:45.114362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.114336 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.170464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.170441 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x"] Nov 03 00:34:45.174399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.174370 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.174742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.174720 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgx5\" (UniqueName: \"kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5\") pod \"payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.217941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.217909 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f2d039\"" Nov 03 00:34:45.262792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.218332 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f2d039-dockercfg-8k9r2\"" Nov 03 00:34:45.310091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.302875 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.310091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.303305 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgx5\" (UniqueName: \"kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5\") pod \"payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.310091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.304694 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.318569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.318508 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54"] Nov 03 00:34:45.324199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.324170 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.328799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.328770 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54"] Nov 03 00:34:45.361523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.361419 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgx5\" (UniqueName: \"kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5\") pod \"payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.428326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.428298 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:45.507676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.507620 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.507997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.507977 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzv9q\" (UniqueName: \"kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q\") pod \"payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.609073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.609044 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.609240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.609229 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzv9q\" (UniqueName: \"kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q\") pod \"payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.609716 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.609698 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.665612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.665588 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzv9q\" (UniqueName: \"kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q\") pod \"payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:45.821285 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:34:45.821230 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd3250bf_17b8_4836_9129_11f3809e35b6.slice/crio-9bc7d469bd9a45deb80062648f4b088bcf23715f23551e24d7c368ff06074693 WatchSource:0}: Error finding container 9bc7d469bd9a45deb80062648f4b088bcf23715f23551e24d7c368ff06074693: Status 404 returned error can't find the container with id 9bc7d469bd9a45deb80062648f4b088bcf23715f23551e24d7c368ff06074693 Nov 03 00:34:45.835874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.834180 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x"] Nov 03 00:34:45.895542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.895510 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:34:45.921164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921140 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921188 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921249 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921303 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921385 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921423 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.921578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921517 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921563 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921608 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.921720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921685 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:45.921720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921703 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.921720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921715 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token\") pod \"dc275659-89d6-429c-8f7b-65976b66f0c9\" (UID: \"dc275659-89d6-429c-8f7b-65976b66f0c9\") " Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.921980 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.922191 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.922370 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.922388 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc275659-89d6-429c-8f7b-65976b66f0c9-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.922402 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.922414 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.922426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.922428 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:45.925323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.925288 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.925538 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.925513 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:34:45.926046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.926024 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "kube-api-access-lrx7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:45.932960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.932927 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy" pod "dc275659-89d6-429c-8f7b-65976b66f0c9" (UID: "dc275659-89d6-429c-8f7b-65976b66f0c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:45.944748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.944728 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:34:45.954116 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:45.954095 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.022983 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023028 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023150 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023188 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023254 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023301 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023338 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023401 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.023465 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket\") pod \"b487d783-1665-4742-ab58-ac8ed1cb2b43\" (UID: \"b487d783-1665-4742-ab58-ac8ed1cb2b43\") " Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.024098 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.024120 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.024135 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrx7m\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-kube-api-access-lrx7m\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.024148 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc275659-89d6-429c-8f7b-65976b66f0c9-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.024578 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.028075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.027917 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.028597 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.033794 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.034148 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.035277 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.035672 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "kube-api-access-ql75d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.044147 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:34:46.046508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.044259 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert" pod "b487d783-1665-4742-ab58-ac8ed1cb2b43" (UID: "b487d783-1665-4742-ab58-ac8ed1cb2b43"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:46.125489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125460 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125489 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125498 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125506 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b487d783-1665-4742-ab58-ac8ed1cb2b43-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125516 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125524 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125533 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b487d783-1665-4742-ab58-ac8ed1cb2b43-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125542 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql75d\" (UniqueName: \"kubernetes.io/projected/b487d783-1665-4742-ab58-ac8ed1cb2b43-kube-api-access-ql75d\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.125887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.125550 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b487d783-1665-4742-ab58-ac8ed1cb2b43-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:46.333792 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:34:46.333758 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945471fa_324a_4df9_9d11_381caf2aa686.slice/crio-ff79abc0a345f5c0d07056376ea7048d2f468c575da6a69f61b7fb40b27e654b WatchSource:0}: Error finding container ff79abc0a345f5c0d07056376ea7048d2f468c575da6a69f61b7fb40b27e654b: Status 404 returned error can't find the container with id ff79abc0a345f5c0d07056376ea7048d2f468c575da6a69f61b7fb40b27e654b Nov 03 00:34:46.333909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.333892 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54"] Nov 03 00:34:46.384396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.384290 5144 generic.go:356] "Generic (PLEG): container finished" podID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerID="af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f" exitCode=0 Nov 03 00:34:46.384396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.384361 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" Nov 03 00:34:46.384396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.384369 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" event={"ID":"dc275659-89d6-429c-8f7b-65976b66f0c9","Type":"ContainerDied","Data":"af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f"} Nov 03 00:34:46.384603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.384421 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d" event={"ID":"dc275659-89d6-429c-8f7b-65976b66f0c9","Type":"ContainerDied","Data":"c77080e4076146385ddab574e56b8bfe31ae85e0e0460c49b33bd311863b02c1"} Nov 03 00:34:46.384603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.384443 5144 scope.go:122] "RemoveContainer" containerID="af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f" Nov 03 00:34:46.386791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.386770 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" event={"ID":"dd3250bf-17b8-4836-9129-11f3809e35b6","Type":"ContainerStarted","Data":"9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b"} Nov 03 00:34:46.386889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.386817 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" event={"ID":"dd3250bf-17b8-4836-9129-11f3809e35b6","Type":"ContainerStarted","Data":"9bc7d469bd9a45deb80062648f4b088bcf23715f23551e24d7c368ff06074693"} Nov 03 00:34:46.386889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.386858 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:46.388437 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.388416 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" event={"ID":"945471fa-324a-4df9-9d11-381caf2aa686","Type":"ContainerStarted","Data":"ff79abc0a345f5c0d07056376ea7048d2f468c575da6a69f61b7fb40b27e654b"} Nov 03 00:34:46.390188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.390166 5144 generic.go:356] "Generic (PLEG): container finished" podID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerID="d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57" exitCode=0 Nov 03 00:34:46.390286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.390196 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" Nov 03 00:34:46.390286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.390249 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" event={"ID":"b487d783-1665-4742-ab58-ac8ed1cb2b43","Type":"ContainerDied","Data":"d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57"} Nov 03 00:34:46.390384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.390300 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b" event={"ID":"b487d783-1665-4742-ab58-ac8ed1cb2b43","Type":"ContainerDied","Data":"51c86fff75997ff3e7f85200db147a42b43ecb876dbe094f47868d7a97ddfc30"} Nov 03 00:34:46.410801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.410776 5144 scope.go:122] "RemoveContainer" containerID="2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677" Nov 03 00:34:46.429935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.429903 5144 scope.go:122] "RemoveContainer" containerID="af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f" Nov 03 00:34:46.430439 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:34:46.430411 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f\": container with ID starting with af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f not found: ID does not exist" containerID="af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f" Nov 03 00:34:46.430534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.430491 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f"} err="failed to get container status \"af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f\": rpc error: code = NotFound desc = could not find container \"af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f\": container with ID starting with af1b368d652c77bc9e95030780c201ed610a32c82ba27da2b88ad02c6226049f not found: ID does not exist" Nov 03 00:34:46.430534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.430518 5144 scope.go:122] "RemoveContainer" containerID="2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677" Nov 03 00:34:46.431113 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:34:46.431091 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677\": container with ID starting with 2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677 not found: ID does not exist" containerID="2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677" Nov 03 00:34:46.431194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.431119 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677"} err="failed to get container status \"2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677\": rpc error: code = NotFound desc = could not find container \"2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677\": container with ID starting with 2af524feebf48fb254edd398825efe818688dfd2c577d07639a23fb934f0e677 not found: ID does not exist" Nov 03 00:34:46.431194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.431137 5144 scope.go:122] "RemoveContainer" containerID="d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57" Nov 03 00:34:46.441853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.441831 5144 scope.go:122] "RemoveContainer" containerID="d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57" Nov 03 00:34:46.442123 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:34:46.442105 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57\": container with ID starting with d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57 not found: ID does not exist" containerID="d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57" Nov 03 00:34:46.442254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.442131 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57"} err="failed to get container status \"d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57\": rpc error: code = NotFound desc = could not find container \"d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57\": container with ID starting with d180f1b82390532c15789164a2cd63592874a1d860c4362626856a47ec33dc57 not found: ID does not exist" Nov 03 00:34:46.553080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.553041 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b"] Nov 03 00:34:46.556535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.556509 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-6tv8b"] Nov 03 00:34:46.665364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.665279 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d"] Nov 03 00:34:46.668659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.668622 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-cb65f3a5-openshift-default-6c6f487df-5966d"] Nov 03 00:34:46.780369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:46.780316 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" podStartSLOduration=1.780301884 podStartE2EDuration="1.780301884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:46.776661737 +0000 UTC m=+1776.922356194" watchObservedRunningTime="2026-11-03 00:34:46.780301884 +0000 UTC m=+1776.925996344" Nov 03 00:34:47.398624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:47.398592 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" event={"ID":"945471fa-324a-4df9-9d11-381caf2aa686","Type":"ContainerStarted","Data":"1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9"} Nov 03 00:34:47.399092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:47.398810 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:47.514475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:47.514430 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" podStartSLOduration=2.514416867 podStartE2EDuration="2.514416867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:47.512523256 +0000 UTC m=+1777.658217713" watchObservedRunningTime="2026-11-03 00:34:47.514416867 +0000 UTC m=+1777.660111322" Nov 03 00:34:47.997315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:47.997283 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" path="/var/lib/kubelet/pods/b487d783-1665-4742-ab58-ac8ed1cb2b43/volumes" Nov 03 00:34:47.997761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:47.997745 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" path="/var/lib/kubelet/pods/dc275659-89d6-429c-8f7b-65976b66f0c9/volumes" Nov 03 00:34:49.712011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.711983 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp"] Nov 03 00:34:49.712662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712625 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" Nov 03 00:34:49.712792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712658 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" Nov 03 00:34:49.712792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712682 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" Nov 03 00:34:49.712792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712687 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" Nov 03 00:34:49.712901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712814 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" Nov 03 00:34:49.712901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712830 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b487d783-1665-4742-ab58-ac8ed1cb2b43" containerName="istio-proxy" Nov 03 00:34:49.712901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.712837 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" Nov 03 00:34:49.731169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.731147 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.772566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.772542 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp"] Nov 03 00:34:49.775366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.775345 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.775514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.775458 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbqn2\" (UniqueName: \"kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.775564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.775515 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.817383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.817358 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:34:49.817383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.817358 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:34:49.877056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.877027 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbqn2\" (UniqueName: \"kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.877251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.877099 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.877319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.877267 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.878015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.877989 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.880182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.880156 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:49.933907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:49.933879 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbqn2\" (UniqueName: \"kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2\") pod \"maas-api-revoke-sub-70b82998-49prp\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:50.047157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:50.047120 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:50.425559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:50.425527 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp"] Nov 03 00:34:50.426252 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:34:50.426228 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4361e5c6_f389_4d14_b71c_93e3bd82ba32.slice/crio-db294d8a9911e3e754e573d149358bcb8ee1c3a49dacb3861bd86461a8118ee9 WatchSource:0}: Error finding container db294d8a9911e3e754e573d149358bcb8ee1c3a49dacb3861bd86461a8118ee9: Status 404 returned error can't find the container with id db294d8a9911e3e754e573d149358bcb8ee1c3a49dacb3861bd86461a8118ee9 Nov 03 00:34:51.445726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:51.445304 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" event={"ID":"4361e5c6-f389-4d14-b71c-93e3bd82ba32","Type":"ContainerStarted","Data":"af32dcdc7dcbfcca3aa5101201f4c3ef18145ae244db8073337f094b76eb17ea"} Nov 03 00:34:51.445726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:51.445355 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" event={"ID":"4361e5c6-f389-4d14-b71c-93e3bd82ba32","Type":"ContainerStarted","Data":"db294d8a9911e3e754e573d149358bcb8ee1c3a49dacb3861bd86461a8118ee9"} Nov 03 00:34:51.560267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:51.560214 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" podStartSLOduration=2.560200725 podStartE2EDuration="2.560200725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:51.558047243 +0000 UTC m=+1781.703741730" watchObservedRunningTime="2026-11-03 00:34:51.560200725 +0000 UTC m=+1781.705895180" Nov 03 00:34:52.456222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:52.456185 5144 generic.go:356] "Generic (PLEG): container finished" podID="4361e5c6-f389-4d14-b71c-93e3bd82ba32" containerID="af32dcdc7dcbfcca3aa5101201f4c3ef18145ae244db8073337f094b76eb17ea" exitCode=0 Nov 03 00:34:52.456671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:52.456275 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" event={"ID":"4361e5c6-f389-4d14-b71c-93e3bd82ba32","Type":"ContainerDied","Data":"af32dcdc7dcbfcca3aa5101201f4c3ef18145ae244db8073337f094b76eb17ea"} Nov 03 00:34:53.676575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.676542 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:53.740439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.740350 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2\" (UniqueName: \"kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2\") pod \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " Nov 03 00:34:53.740439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.740422 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca\") pod \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " Nov 03 00:34:53.740728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.740476 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca\") pod \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\" (UID: \"4361e5c6-f389-4d14-b71c-93e3bd82ba32\") " Nov 03 00:34:53.741136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.741111 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca" pod "4361e5c6-f389-4d14-b71c-93e3bd82ba32" (UID: "4361e5c6-f389-4d14-b71c-93e3bd82ba32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:53.741548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.741529 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:53.747817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.747783 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2" pod "4361e5c6-f389-4d14-b71c-93e3bd82ba32" (UID: "4361e5c6-f389-4d14-b71c-93e3bd82ba32"). InnerVolumeSpecName "kube-api-access-pbqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:53.759824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.759765 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca" pod "4361e5c6-f389-4d14-b71c-93e3bd82ba32" (UID: "4361e5c6-f389-4d14-b71c-93e3bd82ba32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:34:53.847239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.847179 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbqn2\" (UniqueName: \"kubernetes.io/projected/4361e5c6-f389-4d14-b71c-93e3bd82ba32-kube-api-access-pbqn2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:53.847713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:53.847689 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4361e5c6-f389-4d14-b71c-93e3bd82ba32-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:54.496702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:54.496604 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" Nov 03 00:34:54.496912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:54.496622 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp" event={"ID":"4361e5c6-f389-4d14-b71c-93e3bd82ba32","Type":"ContainerDied","Data":"db294d8a9911e3e754e573d149358bcb8ee1c3a49dacb3861bd86461a8118ee9"} Nov 03 00:34:54.496912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:54.496815 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db294d8a9911e3e754e573d149358bcb8ee1c3a49dacb3861bd86461a8118ee9" Nov 03 00:34:56.032124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.032080 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8"] Nov 03 00:34:56.032685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.032289 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" podUID="3b55ffcb-0693-42f3-a605-142c3d026224" containerName="payload-processing" containerID="cri-o://79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3" gracePeriod=30 Nov 03 00:34:56.325656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.325604 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:56.416331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.416299 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume\") pod \"3b55ffcb-0693-42f3-a605-142c3d026224\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " Nov 03 00:34:56.416571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.416414 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff\" (UniqueName: \"kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff\") pod \"3b55ffcb-0693-42f3-a605-142c3d026224\" (UID: \"3b55ffcb-0693-42f3-a605-142c3d026224\") " Nov 03 00:34:56.416759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.416723 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume" pod "3b55ffcb-0693-42f3-a605-142c3d026224" (UID: "3b55ffcb-0693-42f3-a605-142c3d026224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:34:56.417035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.417016 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b55ffcb-0693-42f3-a605-142c3d026224-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:56.418727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.418700 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff" pod "3b55ffcb-0693-42f3-a605-142c3d026224" (UID: "3b55ffcb-0693-42f3-a605-142c3d026224"). InnerVolumeSpecName "kube-api-access-xj5ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:34:56.516746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.516711 5144 generic.go:356] "Generic (PLEG): container finished" podID="3b55ffcb-0693-42f3-a605-142c3d026224" containerID="79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3" exitCode=0 Nov 03 00:34:56.516890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.516764 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" Nov 03 00:34:56.516890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.516774 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" event={"ID":"3b55ffcb-0693-42f3-a605-142c3d026224","Type":"ContainerDied","Data":"79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3"} Nov 03 00:34:56.516890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.516849 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8" event={"ID":"3b55ffcb-0693-42f3-a605-142c3d026224","Type":"ContainerDied","Data":"ef04350d18b4ffbc529b9436c4235499d22b1d16751bfe69a4068069554db6cf"} Nov 03 00:34:56.516890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.516883 5144 scope.go:122] "RemoveContainer" containerID="79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3" Nov 03 00:34:56.518044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.518006 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xj5ff\" (UniqueName: \"kubernetes.io/projected/3b55ffcb-0693-42f3-a605-142c3d026224-kube-api-access-xj5ff\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:34:56.554873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.541778 5144 scope.go:122] "RemoveContainer" containerID="79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3" Nov 03 00:34:56.571717 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:34:56.569193 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3\": container with ID starting with 79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3 not found: ID does not exist" containerID="79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3" Nov 03 00:34:56.571717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.569245 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3"} err="failed to get container status \"79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3\": rpc error: code = NotFound desc = could not find container \"79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3\": container with ID starting with 79f294751b59d56d6c79f8bd629fbad9a5a8c65291d7725e6d048e7706429de3 not found: ID does not exist" Nov 03 00:34:56.701939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.701905 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8"] Nov 03 00:34:56.705563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:56.705541 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-8c9581-695d6687bd-294t8"] Nov 03 00:34:57.404491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:57.404462 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:34:57.997795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:57.997760 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b55ffcb-0693-42f3-a605-142c3d026224" path="/var/lib/kubelet/pods/3b55ffcb-0693-42f3-a605-142c3d026224/volumes" Nov 03 00:34:58.412573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:58.412538 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:34:58.722588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:58.722501 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp"] Nov 03 00:34:58.722794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:58.722766 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" containerID="cri-o://7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c" gracePeriod=30 Nov 03 00:34:59.049073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:59.049043 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2"] Nov 03 00:34:59.052377 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:59.052351 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-294ca089-d9qn2"] Nov 03 00:34:59.997393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:34:59.997356 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11c0bb11-ddfc-4022-bed0-ff0ebaf09393" path="/var/lib/kubelet/pods/11c0bb11-ddfc-4022-bed0-ff0ebaf09393/volumes" Nov 03 00:35:01.275462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:01.275382 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:35:01.275855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:01.275458 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:35:03.696475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:03.696443 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h"] Nov 03 00:35:03.696945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:03.696654 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" containerID="cri-o://019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd" gracePeriod=30 Nov 03 00:35:04.104467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.104438 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:35:04.185762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185732 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.185937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185796 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.185937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185837 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.185937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185869 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.185937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185895 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.186138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185957 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.186138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.185994 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.186138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.186027 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.186279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.186256 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs\") pod \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\" (UID: \"aacca85a-c92c-4051-84ad-a819ad5ff7ea\") " Nov 03 00:35:04.187231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.187187 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:04.187404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.187256 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:04.187689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.187623 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:04.187976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.187668 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:04.187976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.187885 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:04.188795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.188766 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:35:04.189101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.189074 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:04.189699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.189677 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:04.190085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.190068 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5" pod "aacca85a-c92c-4051-84ad-a819ad5ff7ea" (UID: "aacca85a-c92c-4051-84ad-a819ad5ff7ea"). InnerVolumeSpecName "kube-api-access-64ff5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:04.287647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287612 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287664 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287675 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64ff5\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-kube-api-access-64ff5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287686 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287694 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287702 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287709 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287718 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aacca85a-c92c-4051-84ad-a819ad5ff7ea-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.287814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.287726 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aacca85a-c92c-4051-84ad-a819ad5ff7ea-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:04.645911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.645825 5144 generic.go:356] "Generic (PLEG): container finished" podID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerID="7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c" exitCode=0 Nov 03 00:35:04.645911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.645877 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" Nov 03 00:35:04.646164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.645912 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" event={"ID":"aacca85a-c92c-4051-84ad-a819ad5ff7ea","Type":"ContainerDied","Data":"7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c"} Nov 03 00:35:04.646164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.645980 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp" event={"ID":"aacca85a-c92c-4051-84ad-a819ad5ff7ea","Type":"ContainerDied","Data":"9c5c23ee0899610aa1909cb2b2edccfd4612c24d259425feed90a349d58f64c8"} Nov 03 00:35:04.646164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.646011 5144 scope.go:122] "RemoveContainer" containerID="7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c" Nov 03 00:35:04.660689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.660662 5144 scope.go:122] "RemoveContainer" containerID="7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c" Nov 03 00:35:04.660977 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:04.660952 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c\": container with ID starting with 7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c not found: ID does not exist" containerID="7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c" Nov 03 00:35:04.661068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.660985 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c"} err="failed to get container status \"7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c\": rpc error: code = NotFound desc = could not find container \"7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c\": container with ID starting with 7d569e453918a11733ddf4217759b29b3a2f84b4f8f13043122f9b1e3eec875c not found: ID does not exist" Nov 03 00:35:04.810250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.810223 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp"] Nov 03 00:35:04.859823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:04.859790 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-jfwkp"] Nov 03 00:35:06.063874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:06.063836 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" path="/var/lib/kubelet/pods/aacca85a-c92c-4051-84ad-a819ad5ff7ea/volumes" Nov 03 00:35:08.971368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:08.971315 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:35:09.059110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059072 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059127 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059187 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059281 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059370 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059496 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059414 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059496 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059448 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059496 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059545 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token\") pod \"f3b98f61-6633-4f03-b087-eb7645e39aba\" (UID: \"f3b98f61-6633-4f03-b087-eb7645e39aba\") " Nov 03 00:35:09.059763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059591 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:09.059763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059692 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:09.059763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059723 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:09.059935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.059895 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:09.060058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.060035 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:09.060328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.060304 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3b98f61-6633-4f03-b087-eb7645e39aba-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.060396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.060334 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.060396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.060350 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.060396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.060365 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.060396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.060378 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.061580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.061552 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:35:09.061760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.061709 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "kube-api-access-dtzh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:09.061902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.061884 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:35:09.062026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.062009 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token" pod "f3b98f61-6633-4f03-b087-eb7645e39aba" (UID: "f3b98f61-6633-4f03-b087-eb7645e39aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:09.161140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.161099 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.161371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.161344 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.161371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.161371 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3b98f61-6633-4f03-b087-eb7645e39aba-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.161541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.161386 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtzh4\" (UniqueName: \"kubernetes.io/projected/f3b98f61-6633-4f03-b087-eb7645e39aba-kube-api-access-dtzh4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:09.712433 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.712393 5144 generic.go:356] "Generic (PLEG): container finished" podID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerID="019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd" exitCode=0 Nov 03 00:35:09.712661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.712438 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" Nov 03 00:35:09.712661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.712477 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" event={"ID":"f3b98f61-6633-4f03-b087-eb7645e39aba","Type":"ContainerDied","Data":"019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd"} Nov 03 00:35:09.712661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.712539 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h" event={"ID":"f3b98f61-6633-4f03-b087-eb7645e39aba","Type":"ContainerDied","Data":"6b32dae6467c144dbedcc797a335bfaae7e35a5a6ecf20e8536263f9c6fdd5ff"} Nov 03 00:35:09.712661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.712569 5144 scope.go:122] "RemoveContainer" containerID="019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd" Nov 03 00:35:09.724579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.724565 5144 scope.go:122] "RemoveContainer" containerID="d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820" Nov 03 00:35:09.734109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.734083 5144 scope.go:122] "RemoveContainer" containerID="019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd" Nov 03 00:35:09.734377 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:09.734348 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd\": container with ID starting with 019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd not found: ID does not exist" containerID="019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd" Nov 03 00:35:09.734434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.734383 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd"} err="failed to get container status \"019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd\": rpc error: code = NotFound desc = could not find container \"019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd\": container with ID starting with 019dd10749393f755c8256d0b171be2f4cd0c63c619abcf3de3f104fb6e501dd not found: ID does not exist" Nov 03 00:35:09.734434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.734401 5144 scope.go:122] "RemoveContainer" containerID="d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820" Nov 03 00:35:09.734676 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:09.734660 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820\": container with ID starting with d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820 not found: ID does not exist" containerID="d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820" Nov 03 00:35:09.734738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.734681 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820"} err="failed to get container status \"d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820\": rpc error: code = NotFound desc = could not find container \"d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820\": container with ID starting with d98901f94bc85bbac607197639b97b824c035278286671898bbaccfbe18ce820 not found: ID does not exist" Nov 03 00:35:09.926942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.926916 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h"] Nov 03 00:35:09.976353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.976287 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-8c9581-openshift-default-5d48fb6489-kkb7h"] Nov 03 00:35:09.996918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:09.996896 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" path="/var/lib/kubelet/pods/f3b98f61-6633-4f03-b087-eb7645e39aba/volumes" Nov 03 00:35:10.421263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:10.421242 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:35:10.423889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:10.423867 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:35:11.558946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:11.558915 5144 scope.go:122] "RemoveContainer" containerID="a2f327bcadd7a2590f077f0f84731825275a85be19d08bf5ffcf7bd29d805216" Nov 03 00:35:12.155596 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:12.155563 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:35:13.660823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.660784 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k"] Nov 03 00:35:13.661457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661442 5144 cpu_manager.go:414] "containerMap: removing container" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" Nov 03 00:35:13.661495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661459 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" Nov 03 00:35:13.661495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661490 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4361e5c6-f389-4d14-b71c-93e3bd82ba32" containerName="revoke-keys" Nov 03 00:35:13.661495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661495 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4361e5c6-f389-4d14-b71c-93e3bd82ba32" containerName="revoke-keys" Nov 03 00:35:13.661577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661512 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" Nov 03 00:35:13.661577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661517 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" Nov 03 00:35:13.661577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661528 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" Nov 03 00:35:13.661577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661532 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc275659-89d6-429c-8f7b-65976b66f0c9" containerName="istio-proxy" Nov 03 00:35:13.661577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661544 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3b55ffcb-0693-42f3-a605-142c3d026224" containerName="payload-processing" Nov 03 00:35:13.661577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661548 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b55ffcb-0693-42f3-a605-142c3d026224" containerName="payload-processing" Nov 03 00:35:13.661803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661645 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" Nov 03 00:35:13.661803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661656 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" Nov 03 00:35:13.661803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661663 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b55ffcb-0693-42f3-a605-142c3d026224" containerName="payload-processing" Nov 03 00:35:13.661803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661669 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacca85a-c92c-4051-84ad-a819ad5ff7ea" containerName="istio-proxy" Nov 03 00:35:13.661803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.661680 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4361e5c6-f389-4d14-b71c-93e3bd82ba32" containerName="revoke-keys" Nov 03 00:35:13.665567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.665545 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.734851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.734821 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.734892 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.734952 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8frpv\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.735019 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.735077 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.735118 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.735143 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.735204 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.735270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.735236 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.796443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.796395 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k"] Nov 03 00:35:13.836395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836351 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.836569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836442 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.836569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836473 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8frpv\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.836569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836501 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.836569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836537 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.836826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836579 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.836826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.836610 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.837151 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.837127 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.837328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.837309 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.837785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.837764 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.837994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.837972 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.838617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.838597 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.838769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.838700 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.838994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.838972 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.840221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.840197 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.844894 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.844868 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.925169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.925057 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.930690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.930030 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8frpv\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-bln8k\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:13.978877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:13.978850 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:14.425781 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.425748 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k"] Nov 03 00:35:14.430372 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:35:14.430332 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8404df7_e0e6_4fb9_b3c1_2992a602e0ee.slice/crio-0fc3fc5710f6302745041fe3b2d00556f0e5e38844bf84ffcd2247baf6519970 WatchSource:0}: Error finding container 0fc3fc5710f6302745041fe3b2d00556f0e5e38844bf84ffcd2247baf6519970: Status 404 returned error can't find the container with id 0fc3fc5710f6302745041fe3b2d00556f0e5e38844bf84ffcd2247baf6519970 Nov 03 00:35:14.433301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.433268 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.433385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.433333 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.433385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.433364 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:35:14.753837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.753808 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" event={"ID":"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee","Type":"ContainerStarted","Data":"23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72"} Nov 03 00:35:14.754254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.753862 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" event={"ID":"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee","Type":"ContainerStarted","Data":"0fc3fc5710f6302745041fe3b2d00556f0e5e38844bf84ffcd2247baf6519970"} Nov 03 00:35:14.876532 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.876478 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podStartSLOduration=1.876459493 podStartE2EDuration="1.876459493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:14.868517212 +0000 UTC m=+1805.014211695" watchObservedRunningTime="2026-11-03 00:35:14.876459493 +0000 UTC m=+1805.022153953" Nov 03 00:35:14.982847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:14.982799 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:15.981092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:15.981054 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 00:35:15.981471 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:15.981107 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 00:35:16.492330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.492298 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn"] Nov 03 00:35:16.492991 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.492978 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" Nov 03 00:35:16.493044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.492993 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3b98f61-6633-4f03-b087-eb7645e39aba" containerName="istio-proxy" Nov 03 00:35:16.495481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.495467 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.547817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.547790 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-cb486a\"" Nov 03 00:35:16.609661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.590094 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.609661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.590196 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qct7k\" (UniqueName: \"kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k\") pod \"payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.707657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.658073 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-cb486a-dockercfg-pkhrg\"" Nov 03 00:35:16.717659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.713579 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.745915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.744366 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qct7k\" (UniqueName: \"kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k\") pod \"payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.745915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.719954 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.745915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.719158 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn"] Nov 03 00:35:16.745915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.744601 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s"] Nov 03 00:35:16.758657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.757028 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:16.781324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.780296 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s"] Nov 03 00:35:16.804015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.803988 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qct7k\" (UniqueName: \"kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k\") pod \"payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.808602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.808582 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:16.845419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.845386 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdnj\" (UniqueName: \"kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj\") pod \"payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:16.845898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.845866 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:16.950211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.949763 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:16.950211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.949822 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdnj\" (UniqueName: \"kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj\") pod \"payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:16.951484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.951456 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:16.980003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.979976 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 00:35:16.980229 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:16.980203 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 00:35:17.008566 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.008546 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdnj\" (UniqueName: \"kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj\") pod \"payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:17.088744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.088713 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:17.246748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.245747 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn"] Nov 03 00:35:17.490668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.490623 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s"] Nov 03 00:35:17.490922 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:35:17.490897 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbadebde3_1cd7_4860_a18c_3a4be1f02ed7.slice/crio-e455045778eb8c4f0710dffd5e2fbf4bfae22da511b71cc519b9c758df6b16f6 WatchSource:0}: Error finding container e455045778eb8c4f0710dffd5e2fbf4bfae22da511b71cc519b9c758df6b16f6: Status 404 returned error can't find the container with id e455045778eb8c4f0710dffd5e2fbf4bfae22da511b71cc519b9c758df6b16f6 Nov 03 00:35:17.799494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.796208 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" event={"ID":"1205ebf2-392b-46ee-a8a3-31d7f8b29739","Type":"ContainerStarted","Data":"bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1"} Nov 03 00:35:17.799494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.796312 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" event={"ID":"1205ebf2-392b-46ee-a8a3-31d7f8b29739","Type":"ContainerStarted","Data":"bff3b8b0673de3e1e8968c02ac78f6abc0b5cf99d0916835ee04d7a314fc6fb9"} Nov 03 00:35:17.799494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.797614 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:17.803708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.803677 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" event={"ID":"badebde3-1cd7-4860-a18c-3a4be1f02ed7","Type":"ContainerStarted","Data":"5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007"} Nov 03 00:35:17.803846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.803719 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" event={"ID":"badebde3-1cd7-4860-a18c-3a4be1f02ed7","Type":"ContainerStarted","Data":"e455045778eb8c4f0710dffd5e2fbf4bfae22da511b71cc519b9c758df6b16f6"} Nov 03 00:35:17.803846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.803819 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:17.910805 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.910745 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" podStartSLOduration=1.910733032 podStartE2EDuration="1.910733032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:17.910041599 +0000 UTC m=+1808.055736056" watchObservedRunningTime="2026-11-03 00:35:17.910733032 +0000 UTC m=+1808.056427487" Nov 03 00:35:17.980048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.980013 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 00:35:17.980225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:17.980074 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 00:35:18.024663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:18.024248 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" podStartSLOduration=2.024230545 podStartE2EDuration="2.024230545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:18.023022653 +0000 UTC m=+1808.168717133" watchObservedRunningTime="2026-11-03 00:35:18.024230545 +0000 UTC m=+1808.169925002" Nov 03 00:35:18.980030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:18.979988 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 00:35:18.980287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:18.980055 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 00:35:19.979739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:19.979711 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 00:35:19.980122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:19.979758 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 00:35:20.979619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:20.979590 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" start-of-body= Nov 03 00:35:20.979823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:20.979659 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.182:15021/healthz/ready\": dial tcp 10.132.0.182:15021: connect: connection refused" Nov 03 00:35:22.605068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:22.605028 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:22.605733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:22.605710 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:22.606844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:22.606824 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:35:28.814288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:28.814260 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:35:29.856984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:29.856953 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:35:33.728835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:33.728779 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" podUID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.139:8000/ready\": context deadline exceeded" Nov 03 00:35:42.155045 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:42.155006 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:35:50.200949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:50.200918 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7"] Nov 03 00:35:50.312297 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:50.312264 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np"] Nov 03 00:35:55.201443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.201404 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" podUID="a8e09516-4b77-4e8e-8174-8c27a69aa96d" containerName="payload-processing" containerID="cri-o://50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811" gracePeriod=25 Nov 03 00:35:55.313476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.313434 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" podUID="9187ddba-9bec-4b7d-b081-482223b24fcf" containerName="payload-pre-processing" containerID="cri-o://5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0" gracePeriod=25 Nov 03 00:35:55.501729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.501702 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:35:55.627108 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.627079 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:35:55.655026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.654307 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume\") pod \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " Nov 03 00:35:55.655026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.654504 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh\" (UniqueName: \"kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh\") pod \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\" (UID: \"a8e09516-4b77-4e8e-8174-8c27a69aa96d\") " Nov 03 00:35:55.655026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.654758 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume" pod "a8e09516-4b77-4e8e-8174-8c27a69aa96d" (UID: "a8e09516-4b77-4e8e-8174-8c27a69aa96d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:55.655366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.655116 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8e09516-4b77-4e8e-8174-8c27a69aa96d-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:55.657232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.657188 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh" pod "a8e09516-4b77-4e8e-8174-8c27a69aa96d" (UID: "a8e09516-4b77-4e8e-8174-8c27a69aa96d"). InnerVolumeSpecName "kube-api-access-44kkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:55.755780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.755748 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume\") pod \"9187ddba-9bec-4b7d-b081-482223b24fcf\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " Nov 03 00:35:55.755976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.755853 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl\" (UniqueName: \"kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl\") pod \"9187ddba-9bec-4b7d-b081-482223b24fcf\" (UID: \"9187ddba-9bec-4b7d-b081-482223b24fcf\") " Nov 03 00:35:55.756420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.756397 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume" pod "9187ddba-9bec-4b7d-b081-482223b24fcf" (UID: "9187ddba-9bec-4b7d-b081-482223b24fcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:35:55.756552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.756509 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44kkh\" (UniqueName: \"kubernetes.io/projected/a8e09516-4b77-4e8e-8174-8c27a69aa96d-kube-api-access-44kkh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:55.759429 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.759403 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl" pod "9187ddba-9bec-4b7d-b081-482223b24fcf" (UID: "9187ddba-9bec-4b7d-b081-482223b24fcf"). InnerVolumeSpecName "kube-api-access-n45kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:35:55.858104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.858057 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9187ddba-9bec-4b7d-b081-482223b24fcf-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:55.858104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:55.858094 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n45kl\" (UniqueName: \"kubernetes.io/projected/9187ddba-9bec-4b7d-b081-482223b24fcf-kube-api-access-n45kl\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:35:56.192907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.192869 5144 generic.go:356] "Generic (PLEG): container finished" podID="a8e09516-4b77-4e8e-8174-8c27a69aa96d" containerID="50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811" exitCode=0 Nov 03 00:35:56.193087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.192918 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" Nov 03 00:35:56.193087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.192953 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" event={"ID":"a8e09516-4b77-4e8e-8174-8c27a69aa96d","Type":"ContainerDied","Data":"50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811"} Nov 03 00:35:56.193087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.193015 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7" event={"ID":"a8e09516-4b77-4e8e-8174-8c27a69aa96d","Type":"ContainerDied","Data":"d6acb4401ae758ab21dda1649f7cc5567246fec7889202bbd7dabebe055647ce"} Nov 03 00:35:56.193087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.193040 5144 scope.go:122] "RemoveContainer" containerID="50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811" Nov 03 00:35:56.198680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.194680 5144 generic.go:356] "Generic (PLEG): container finished" podID="9187ddba-9bec-4b7d-b081-482223b24fcf" containerID="5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0" exitCode=0 Nov 03 00:35:56.198680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.194717 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" Nov 03 00:35:56.198680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.194916 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" event={"ID":"9187ddba-9bec-4b7d-b081-482223b24fcf","Type":"ContainerDied","Data":"5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0"} Nov 03 00:35:56.198680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.194954 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np" event={"ID":"9187ddba-9bec-4b7d-b081-482223b24fcf","Type":"ContainerDied","Data":"a8c331fc65710d9f266dfca1bf058d3717dc6ed475218cba82ba4f0830888f25"} Nov 03 00:35:56.207111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.206902 5144 scope.go:122] "RemoveContainer" containerID="50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811" Nov 03 00:35:56.207367 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:56.207195 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811\": container with ID starting with 50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811 not found: ID does not exist" containerID="50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811" Nov 03 00:35:56.207367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.207219 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811"} err="failed to get container status \"50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811\": rpc error: code = NotFound desc = could not find container \"50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811\": container with ID starting with 50a87cbc6fdc8318f4de5ead9981aad44b785f157947ca73f10ed3396a8a9811 not found: ID does not exist" Nov 03 00:35:56.207367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.207235 5144 scope.go:122] "RemoveContainer" containerID="5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0" Nov 03 00:35:56.259861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.259835 5144 scope.go:122] "RemoveContainer" containerID="5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0" Nov 03 00:35:56.260324 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:35:56.260302 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0\": container with ID starting with 5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0 not found: ID does not exist" containerID="5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0" Nov 03 00:35:56.260410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.260336 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0"} err="failed to get container status \"5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0\": rpc error: code = NotFound desc = could not find container \"5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0\": container with ID starting with 5f72c00a77bf8adfeda45f9f83e684c8016b05fb330b6f8e471e1f95190022c0 not found: ID does not exist" Nov 03 00:35:56.362732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.362702 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np"] Nov 03 00:35:56.366113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.366087 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-eb20df-8c4d74bff-pq2np"] Nov 03 00:35:56.472817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.472784 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7"] Nov 03 00:35:56.480460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:56.480434 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-eb20df-7844f49445-gbtz7"] Nov 03 00:35:57.997343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:57.997312 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9187ddba-9bec-4b7d-b081-482223b24fcf" path="/var/lib/kubelet/pods/9187ddba-9bec-4b7d-b081-482223b24fcf/volumes" Nov 03 00:35:57.997735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:35:57.997723 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e09516-4b77-4e8e-8174-8c27a69aa96d" path="/var/lib/kubelet/pods/a8e09516-4b77-4e8e-8174-8c27a69aa96d/volumes" Nov 03 00:36:01.046662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:01.046619 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x"] Nov 03 00:36:01.096354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:01.096331 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3f14304-wzf6x"] Nov 03 00:36:01.997547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:01.997514 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443a4a34-eb74-4615-a85f-bf35e4d9e6c7" path="/var/lib/kubelet/pods/443a4a34-eb74-4615-a85f-bf35e4d9e6c7/volumes" Nov 03 00:36:06.592851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.592817 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv"] Nov 03 00:36:06.593432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.593419 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a8e09516-4b77-4e8e-8174-8c27a69aa96d" containerName="payload-processing" Nov 03 00:36:06.593479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.593433 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e09516-4b77-4e8e-8174-8c27a69aa96d" containerName="payload-processing" Nov 03 00:36:06.593479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.593466 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9187ddba-9bec-4b7d-b081-482223b24fcf" containerName="payload-pre-processing" Nov 03 00:36:06.593479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.593471 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9187ddba-9bec-4b7d-b081-482223b24fcf" containerName="payload-pre-processing" Nov 03 00:36:06.593576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.593565 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9187ddba-9bec-4b7d-b081-482223b24fcf" containerName="payload-pre-processing" Nov 03 00:36:06.593608 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.593578 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e09516-4b77-4e8e-8174-8c27a69aa96d" containerName="payload-processing" Nov 03 00:36:06.597293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.597270 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.653498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.653474 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv"] Nov 03 00:36:06.692826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.692786 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.693041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.693023 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm662\" (UniqueName: \"kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.698546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.698521 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1a883c\"" Nov 03 00:36:06.698913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.698890 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1a883c-dockercfg-dl2vx\"" Nov 03 00:36:06.793740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.793714 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.793934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.793914 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm662\" (UniqueName: \"kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.794454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.794437 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.855793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.855730 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm662\" (UniqueName: \"kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662\") pod \"payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:06.911341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:06.911315 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:07.289299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.289271 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv"] Nov 03 00:36:07.289478 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:07.289455 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f627b6_06f4_4745_97f1_32ffeda2a032.slice/crio-92187c28f5306661bb4c3a29626b86883d95e3f1415ddf2ba01a239c9c9fa953 WatchSource:0}: Error finding container 92187c28f5306661bb4c3a29626b86883d95e3f1415ddf2ba01a239c9c9fa953: Status 404 returned error can't find the container with id 92187c28f5306661bb4c3a29626b86883d95e3f1415ddf2ba01a239c9c9fa953 Nov 03 00:36:07.398948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.398920 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w"] Nov 03 00:36:07.403235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.403216 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.463090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.463066 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w"] Nov 03 00:36:07.501500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.501478 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltb9x\" (UniqueName: \"kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x\") pod \"payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.501609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.501520 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.605985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.603827 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltb9x\" (UniqueName: \"kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x\") pod \"payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.605985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.603900 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.605985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.604953 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.664352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.664328 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltb9x\" (UniqueName: \"kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x\") pod \"payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:07.716004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:07.715978 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:08.101531 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:08.101502 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c29fecd_d6b1_4d1d_b561_9bcac32b83a7.slice/crio-dccef021162cd977c635d519d2c30042c58a0d91a4f0679d56b68537f0c39aee WatchSource:0}: Error finding container dccef021162cd977c635d519d2c30042c58a0d91a4f0679d56b68537f0c39aee: Status 404 returned error can't find the container with id dccef021162cd977c635d519d2c30042c58a0d91a4f0679d56b68537f0c39aee Nov 03 00:36:08.102580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.102554 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w"] Nov 03 00:36:08.291787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.291750 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" event={"ID":"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7","Type":"ContainerStarted","Data":"1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138"} Nov 03 00:36:08.291948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.291813 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" event={"ID":"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7","Type":"ContainerStarted","Data":"dccef021162cd977c635d519d2c30042c58a0d91a4f0679d56b68537f0c39aee"} Nov 03 00:36:08.291948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.291844 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:08.293686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.293653 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" event={"ID":"06f627b6-06f4-4745-97f1-32ffeda2a032","Type":"ContainerStarted","Data":"f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66"} Nov 03 00:36:08.293823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.293700 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" event={"ID":"06f627b6-06f4-4745-97f1-32ffeda2a032","Type":"ContainerStarted","Data":"92187c28f5306661bb4c3a29626b86883d95e3f1415ddf2ba01a239c9c9fa953"} Nov 03 00:36:08.293823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.293738 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:08.409610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:08.409496 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" podStartSLOduration=1.40948093 podStartE2EDuration="1.40948093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:08.404744319 +0000 UTC m=+1858.550438808" watchObservedRunningTime="2026-11-03 00:36:08.40948093 +0000 UTC m=+1858.555175385" Nov 03 00:36:11.613250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:11.613220 5144 scope.go:122] "RemoveContainer" containerID="bb7668f7ee4ec5c986384fce2df50c766ba6f68a65ca18f1858269e8cb8f40b2" Nov 03 00:36:11.624675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:11.624625 5144 scope.go:122] "RemoveContainer" containerID="10ad300b0c1d719f3e847fe324bfd5fce1940ace4e65632f94477ac8bf08b8c5" Nov 03 00:36:12.155155 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:12.155115 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:36:14.297497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:14.297450 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" podStartSLOduration=8.297437808 podStartE2EDuration="8.297437808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:08.551501205 +0000 UTC m=+1858.697195661" watchObservedRunningTime="2026-11-03 00:36:14.297437808 +0000 UTC m=+1864.443132263" Nov 03 00:36:14.300028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:14.300008 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82"] Nov 03 00:36:14.357282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:14.357241 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" containerID="cri-o://a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d" gracePeriod=30 Nov 03 00:36:17.503057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:17.503018 5144 patch_prober.go:29] interesting pod/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:36:17.503440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:17.503100 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:36:19.304788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.304758 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:36:19.305236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.304830 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:36:19.654915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.654890 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788187 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788277 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788334 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788375 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788401 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788436 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788459 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788588 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.791673 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.788655 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert\") pod \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\" (UID: \"e88c37e5-26f1-453a-8e25-4f288c2b8c24\") " Nov 03 00:36:19.792958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.792683 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:19.792958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.792796 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:19.793203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.793031 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:19.845655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.823751 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:19.846532 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.846490 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:19.872993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.872901 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "kube-api-access-wr6rj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:19.873756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.873713 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:19.873904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.873826 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:36:19.874330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.874301 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy" pod "e88c37e5-26f1-453a-8e25-4f288c2b8c24" (UID: "e88c37e5-26f1-453a-8e25-4f288c2b8c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:19.890974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.890943 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr6rj\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-kube-api-access-wr6rj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.890974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.890970 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.890980 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.890988 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.890999 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.891006 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.891014 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.891022 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:19.891181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:19.891032 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e88c37e5-26f1-453a-8e25-4f288c2b8c24-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:20.417326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.417291 5144 generic.go:356] "Generic (PLEG): container finished" podID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerID="a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d" exitCode=0 Nov 03 00:36:20.417794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.417330 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" event={"ID":"e88c37e5-26f1-453a-8e25-4f288c2b8c24","Type":"ContainerDied","Data":"a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d"} Nov 03 00:36:20.417794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.417359 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" Nov 03 00:36:20.417794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.417433 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82" event={"ID":"e88c37e5-26f1-453a-8e25-4f288c2b8c24","Type":"ContainerDied","Data":"81c97a8619678c08e3915a0824d449b4361de4e85bed559cb3f7d9b6c8fee4cc"} Nov 03 00:36:20.417794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.417461 5144 scope.go:122] "RemoveContainer" containerID="a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d" Nov 03 00:36:20.435311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.435286 5144 scope.go:122] "RemoveContainer" containerID="a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d" Nov 03 00:36:20.435661 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:20.435609 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d\": container with ID starting with a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d not found: ID does not exist" containerID="a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d" Nov 03 00:36:20.435661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.435653 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d"} err="failed to get container status \"a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d\": rpc error: code = NotFound desc = could not find container \"a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d\": container with ID starting with a29f4cd992bb834fe050bc3a8db6ae8e9c2ce250e632e0870c34a1d22c4c0b2d not found: ID does not exist" Nov 03 00:36:20.584060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.584033 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82"] Nov 03 00:36:20.589089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:20.589063 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-lwg82"] Nov 03 00:36:21.997293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:21.997262 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" path="/var/lib/kubelet/pods/e88c37e5-26f1-453a-8e25-4f288c2b8c24/volumes" Nov 03 00:36:28.708212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:28.708175 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776"] Nov 03 00:36:28.708722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:28.708419 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" containerID="cri-o://18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9" gracePeriod=30 Nov 03 00:36:29.159370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.159327 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx"] Nov 03 00:36:29.160057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.160038 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" Nov 03 00:36:29.160181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.160058 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" Nov 03 00:36:29.160281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.160267 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e88c37e5-26f1-453a-8e25-4f288c2b8c24" containerName="istio-proxy" Nov 03 00:36:29.165396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.165374 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.217450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.217404 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-011885\"" Nov 03 00:36:29.217580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.217506 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-011885-dockercfg-6rhn6\"" Nov 03 00:36:29.220391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.220368 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx"] Nov 03 00:36:29.229607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.229571 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwxtq\" (UniqueName: \"kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq\") pod \"payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.229759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.229652 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.346405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.346365 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwxtq\" (UniqueName: \"kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq\") pod \"payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.347548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.347519 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.348840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.348818 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.377987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.374893 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7"] Nov 03 00:36:29.381655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.380319 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.381655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.381223 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7"] Nov 03 00:36:29.405670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.405589 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwxtq\" (UniqueName: \"kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq\") pod \"payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.448777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.448707 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjm49\" (UniqueName: \"kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49\") pod \"payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.448914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.448858 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.481328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.481304 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:29.550393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.550340 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjm49\" (UniqueName: \"kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49\") pod \"payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.550520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.550481 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.551558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.551533 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.609781 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.609721 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjm49\" (UniqueName: \"kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49\") pod \"payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.701275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.701168 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:29.881400 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:29.881374 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx"] Nov 03 00:36:29.881977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:29.881932 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeee5189_ddaf_496f_9627_770f44f39571.slice/crio-a91da3b2e18f0c5a26d8fbcfee18026e2ddd9948dad1712a21a1c5bbdea9da2d WatchSource:0}: Error finding container a91da3b2e18f0c5a26d8fbcfee18026e2ddd9948dad1712a21a1c5bbdea9da2d: Status 404 returned error can't find the container with id a91da3b2e18f0c5a26d8fbcfee18026e2ddd9948dad1712a21a1c5bbdea9da2d Nov 03 00:36:30.079451 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.079423 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7"] Nov 03 00:36:30.080114 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:30.080090 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba1ead1_21a4_491b_8bf3_58a386204e13.slice/crio-0ca8761ad4c3e52e36076edb0cafdd42d3a891e8c2dd17321b073d13028a594a WatchSource:0}: Error finding container 0ca8761ad4c3e52e36076edb0cafdd42d3a891e8c2dd17321b073d13028a594a: Status 404 returned error can't find the container with id 0ca8761ad4c3e52e36076edb0cafdd42d3a891e8c2dd17321b073d13028a594a Nov 03 00:36:30.528496 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.528457 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" event={"ID":"beee5189-ddaf-496f-9627-770f44f39571","Type":"ContainerStarted","Data":"e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3"} Nov 03 00:36:30.528699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.528522 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" event={"ID":"beee5189-ddaf-496f-9627-770f44f39571","Type":"ContainerStarted","Data":"a91da3b2e18f0c5a26d8fbcfee18026e2ddd9948dad1712a21a1c5bbdea9da2d"} Nov 03 00:36:30.528788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.528695 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:30.530447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.530423 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" event={"ID":"9ba1ead1-21a4-491b-8bf3-58a386204e13","Type":"ContainerStarted","Data":"13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2"} Nov 03 00:36:30.530557 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.530460 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" event={"ID":"9ba1ead1-21a4-491b-8bf3-58a386204e13","Type":"ContainerStarted","Data":"0ca8761ad4c3e52e36076edb0cafdd42d3a891e8c2dd17321b073d13028a594a"} Nov 03 00:36:30.530604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.530554 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:30.642459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.642398 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" podStartSLOduration=1.642379555 podStartE2EDuration="1.642379555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:30.640092981 +0000 UTC m=+1880.785787432" watchObservedRunningTime="2026-11-03 00:36:30.642379555 +0000 UTC m=+1880.788074010" Nov 03 00:36:30.766585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:30.766486 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" podStartSLOduration=1.7664636439999999 podStartE2EDuration="1.766463644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:30.757341132 +0000 UTC m=+1880.903035621" watchObservedRunningTime="2026-11-03 00:36:30.766463644 +0000 UTC m=+1880.912158114" Nov 03 00:36:31.339593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.339556 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w"] Nov 03 00:36:31.346951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.346918 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.373996 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.373969 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.374141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.374012 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.374141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.374028 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4wll\" (UniqueName: \"kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.395711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.395690 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w"] Nov 03 00:36:31.469509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.469481 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:36:31.469662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.469516 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:36:31.475147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.475123 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.475269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.475164 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.475269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.475181 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4wll\" (UniqueName: \"kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.475974 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.475790 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.477421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.477398 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.532408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.532378 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4wll\" (UniqueName: \"kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll\") pod \"maas-api-revoke-sub-7ba0573f-gbc6w\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:31.661172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:31.661109 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:32.052785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:32.052757 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w"] Nov 03 00:36:32.052922 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:32.052888 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3270557_ac9c_4c26_80cf_dcf9d00191f1.slice/crio-2e28ba0b8d9195b77708db23fd8f799209cfbe9fc98b66b9c0e5a4936b142c0d WatchSource:0}: Error finding container 2e28ba0b8d9195b77708db23fd8f799209cfbe9fc98b66b9c0e5a4936b142c0d: Status 404 returned error can't find the container with id 2e28ba0b8d9195b77708db23fd8f799209cfbe9fc98b66b9c0e5a4936b142c0d Nov 03 00:36:32.547708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:32.547662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" event={"ID":"c3270557-ac9c-4c26-80cf-dcf9d00191f1","Type":"ContainerStarted","Data":"4e16009f275184ec7928c5ec0fff36cd1df5aaeec06faaac220a03ee5e0d0e62"} Nov 03 00:36:32.548179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:32.547736 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" event={"ID":"c3270557-ac9c-4c26-80cf-dcf9d00191f1","Type":"ContainerStarted","Data":"2e28ba0b8d9195b77708db23fd8f799209cfbe9fc98b66b9c0e5a4936b142c0d"} Nov 03 00:36:32.661061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:32.660969 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" podStartSLOduration=1.660956487 podStartE2EDuration="1.660956487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:32.659318428 +0000 UTC m=+1882.805012920" watchObservedRunningTime="2026-11-03 00:36:32.660956487 +0000 UTC m=+1882.806650942" Nov 03 00:36:33.557289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:33.557255 5144 generic.go:356] "Generic (PLEG): container finished" podID="c3270557-ac9c-4c26-80cf-dcf9d00191f1" containerID="4e16009f275184ec7928c5ec0fff36cd1df5aaeec06faaac220a03ee5e0d0e62" exitCode=0 Nov 03 00:36:33.557703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:33.557337 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" event={"ID":"c3270557-ac9c-4c26-80cf-dcf9d00191f1","Type":"ContainerDied","Data":"4e16009f275184ec7928c5ec0fff36cd1df5aaeec06faaac220a03ee5e0d0e62"} Nov 03 00:36:33.967511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:33.967483 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:36:34.012244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012216 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012311 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012384 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012444 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012472 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012586 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012624 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012643 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.012710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012673 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012735 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy\") pod \"1af72847-0d9a-4cc3-80b8-36689800df6d\" (UID: \"1af72847-0d9a-4cc3-80b8-36689800df6d\") " Nov 03 00:36:34.012868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012786 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.012980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.012874 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.013072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013039 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.013264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013234 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:34.013484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013458 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.013484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013485 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.013701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013500 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.013701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013514 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1af72847-0d9a-4cc3-80b8-36689800df6d-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.013701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.013528 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.014902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.014874 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:34.014999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.014951 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "kube-api-access-sthgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:34.015049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.014995 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:36:34.015306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.015286 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy" pod "1af72847-0d9a-4cc3-80b8-36689800df6d" (UID: "1af72847-0d9a-4cc3-80b8-36689800df6d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:36:34.115289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.115210 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.115289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.115250 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthgk\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-kube-api-access-sthgk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.115289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.115267 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.115289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.115280 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1af72847-0d9a-4cc3-80b8-36689800df6d-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.566971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.566931 5144 generic.go:356] "Generic (PLEG): container finished" podID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerID="18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9" exitCode=0 Nov 03 00:36:34.567410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.566993 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" Nov 03 00:36:34.567410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.567015 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" event={"ID":"1af72847-0d9a-4cc3-80b8-36689800df6d","Type":"ContainerDied","Data":"18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9"} Nov 03 00:36:34.567410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.567095 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776" event={"ID":"1af72847-0d9a-4cc3-80b8-36689800df6d","Type":"ContainerDied","Data":"fe20deb790cc3abeeac2f6a10066c334697c74372225bd894bdd13620f24e38f"} Nov 03 00:36:34.567410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.567126 5144 scope.go:122] "RemoveContainer" containerID="18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9" Nov 03 00:36:34.601166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.601143 5144 scope.go:122] "RemoveContainer" containerID="18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9" Nov 03 00:36:34.604148 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:34.603905 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9\": container with ID starting with 18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9 not found: ID does not exist" containerID="18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9" Nov 03 00:36:34.604275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.604171 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9"} err="failed to get container status \"18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9\": rpc error: code = NotFound desc = could not find container \"18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9\": container with ID starting with 18dccbeed8464367fb4abdef0d4f3ff2091d0024bd182fdbacf749e0a418a0f9 not found: ID does not exist" Nov 03 00:36:34.739026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.738989 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776"] Nov 03 00:36:34.765277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.765249 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:34.786357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.786319 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-9x776"] Nov 03 00:36:34.822780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.822716 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca\") pod \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " Nov 03 00:36:34.822780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.822754 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca\") pod \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " Nov 03 00:36:34.822966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.822865 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll\" (UniqueName: \"kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll\") pod \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\" (UID: \"c3270557-ac9c-4c26-80cf-dcf9d00191f1\") " Nov 03 00:36:34.823087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.823062 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca" pod "c3270557-ac9c-4c26-80cf-dcf9d00191f1" (UID: "c3270557-ac9c-4c26-80cf-dcf9d00191f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:34.823291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.823273 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.825127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.825103 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca" pod "c3270557-ac9c-4c26-80cf-dcf9d00191f1" (UID: "c3270557-ac9c-4c26-80cf-dcf9d00191f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:36:34.825209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.825183 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll" pod "c3270557-ac9c-4c26-80cf-dcf9d00191f1" (UID: "c3270557-ac9c-4c26-80cf-dcf9d00191f1"). InnerVolumeSpecName "kube-api-access-r4wll". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:34.924835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.924799 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4wll\" (UniqueName: \"kubernetes.io/projected/c3270557-ac9c-4c26-80cf-dcf9d00191f1-kube-api-access-r4wll\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:34.924835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:34.924836 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3270557-ac9c-4c26-80cf-dcf9d00191f1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:35.576693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:35.576668 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" Nov 03 00:36:35.576693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:35.576686 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w" event={"ID":"c3270557-ac9c-4c26-80cf-dcf9d00191f1","Type":"ContainerDied","Data":"2e28ba0b8d9195b77708db23fd8f799209cfbe9fc98b66b9c0e5a4936b142c0d"} Nov 03 00:36:35.577245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:35.576733 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e28ba0b8d9195b77708db23fd8f799209cfbe9fc98b66b9c0e5a4936b142c0d" Nov 03 00:36:35.997090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:35.997013 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" path="/var/lib/kubelet/pods/1af72847-0d9a-4cc3-80b8-36689800df6d/volumes" Nov 03 00:36:41.547843 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:41.547804 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:36:41.548376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:41.548155 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:36:42.199654 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:42.199605 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:36:45.039655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:45.028116 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54"] Nov 03 00:36:45.152579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:45.152543 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x"] Nov 03 00:36:50.041291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.041240 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" podUID="945471fa-324a-4df9-9d11-381caf2aa686" containerName="payload-processing" containerID="cri-o://1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9" gracePeriod=25 Nov 03 00:36:50.118825 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:50.118794 5144 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod945471fa_324a_4df9_9d11_381caf2aa686.slice/crio-conmon-1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9.scope\": RecentStats: unable to find data in memory cache]" Nov 03 00:36:50.153927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.153893 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" podUID="dd3250bf-17b8-4836-9129-11f3809e35b6" containerName="payload-pre-processing" containerID="cri-o://9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b" gracePeriod=25 Nov 03 00:36:50.349969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.349939 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:36:50.430595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.426558 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume\") pod \"945471fa-324a-4df9-9d11-381caf2aa686\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " Nov 03 00:36:50.430595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.426753 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q\" (UniqueName: \"kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q\") pod \"945471fa-324a-4df9-9d11-381caf2aa686\" (UID: \"945471fa-324a-4df9-9d11-381caf2aa686\") " Nov 03 00:36:50.431965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.431378 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q" pod "945471fa-324a-4df9-9d11-381caf2aa686" (UID: "945471fa-324a-4df9-9d11-381caf2aa686"). InnerVolumeSpecName "kube-api-access-fzv9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:50.431965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.431852 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume" pod "945471fa-324a-4df9-9d11-381caf2aa686" (UID: "945471fa-324a-4df9-9d11-381caf2aa686"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:50.528822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.528781 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/945471fa-324a-4df9-9d11-381caf2aa686-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:50.528822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.528818 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzv9q\" (UniqueName: \"kubernetes.io/projected/945471fa-324a-4df9-9d11-381caf2aa686-kube-api-access-fzv9q\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:50.568668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.568556 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:36:50.629551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.629519 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume\") pod \"dd3250bf-17b8-4836-9129-11f3809e35b6\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " Nov 03 00:36:50.629726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.629621 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5\" (UniqueName: \"kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5\") pod \"dd3250bf-17b8-4836-9129-11f3809e35b6\" (UID: \"dd3250bf-17b8-4836-9129-11f3809e35b6\") " Nov 03 00:36:50.629938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.629909 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume" pod "dd3250bf-17b8-4836-9129-11f3809e35b6" (UID: "dd3250bf-17b8-4836-9129-11f3809e35b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:36:50.630177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.630159 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd3250bf-17b8-4836-9129-11f3809e35b6-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:50.631878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.631857 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5" pod "dd3250bf-17b8-4836-9129-11f3809e35b6" (UID: "dd3250bf-17b8-4836-9129-11f3809e35b6"). InnerVolumeSpecName "kube-api-access-5bgx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:36:50.734677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.731315 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgx5\" (UniqueName: \"kubernetes.io/projected/dd3250bf-17b8-4836-9129-11f3809e35b6-kube-api-access-5bgx5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:36:50.800155 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.800128 5144 generic.go:356] "Generic (PLEG): container finished" podID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerID="e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017" exitCode=0 Nov 03 00:36:50.800309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.800199 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" event={"ID":"81b79a89-52ca-4440-9874-cc9a78bc0735","Type":"ContainerDied","Data":"e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017"} Nov 03 00:36:50.800665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.800624 5144 scope.go:122] "RemoveContainer" containerID="e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017" Nov 03 00:36:50.801668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.801613 5144 generic.go:356] "Generic (PLEG): container finished" podID="945471fa-324a-4df9-9d11-381caf2aa686" containerID="1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9" exitCode=0 Nov 03 00:36:50.801668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.801666 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" Nov 03 00:36:50.801824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.801787 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" event={"ID":"945471fa-324a-4df9-9d11-381caf2aa686","Type":"ContainerDied","Data":"1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9"} Nov 03 00:36:50.801870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.801854 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54" event={"ID":"945471fa-324a-4df9-9d11-381caf2aa686","Type":"ContainerDied","Data":"ff79abc0a345f5c0d07056376ea7048d2f468c575da6a69f61b7fb40b27e654b"} Nov 03 00:36:50.801909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.801883 5144 scope.go:122] "RemoveContainer" containerID="1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9" Nov 03 00:36:50.802211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.802184 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:50.802312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.802257 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:50.802312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.802293 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:36:50.803449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.803423 5144 generic.go:356] "Generic (PLEG): container finished" podID="dd3250bf-17b8-4836-9129-11f3809e35b6" containerID="9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b" exitCode=0 Nov 03 00:36:50.803553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.803465 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" Nov 03 00:36:50.803553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.803465 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" event={"ID":"dd3250bf-17b8-4836-9129-11f3809e35b6","Type":"ContainerDied","Data":"9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b"} Nov 03 00:36:50.803704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.803578 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x" event={"ID":"dd3250bf-17b8-4836-9129-11f3809e35b6","Type":"ContainerDied","Data":"9bc7d469bd9a45deb80062648f4b088bcf23715f23551e24d7c368ff06074693"} Nov 03 00:36:50.820671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.820656 5144 scope.go:122] "RemoveContainer" containerID="1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9" Nov 03 00:36:50.820992 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:50.820964 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9\": container with ID starting with 1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9 not found: ID does not exist" containerID="1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9" Nov 03 00:36:50.821061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.821002 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9"} err="failed to get container status \"1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9\": rpc error: code = NotFound desc = could not find container \"1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9\": container with ID starting with 1ef1eccd19d0cde818af0b06128593a43ffacbb73caea97136fb618b61c208a9 not found: ID does not exist" Nov 03 00:36:50.821061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.821024 5144 scope.go:122] "RemoveContainer" containerID="9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b" Nov 03 00:36:50.833563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.833542 5144 scope.go:122] "RemoveContainer" containerID="9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b" Nov 03 00:36:50.833938 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:36:50.833913 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b\": container with ID starting with 9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b not found: ID does not exist" containerID="9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b" Nov 03 00:36:50.834033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:50.833947 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b"} err="failed to get container status \"9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b\": rpc error: code = NotFound desc = could not find container \"9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b\": container with ID starting with 9694d65435021da9b1071de4508b04bd8cc606d7f1e1a41016c36727430b1a8b not found: ID does not exist" Nov 03 00:36:51.032504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.032473 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x"] Nov 03 00:36:51.038811 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.038783 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f2d039-57f6dcd855-gf65x"] Nov 03 00:36:51.149079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.148999 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54"] Nov 03 00:36:51.154920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.154896 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f2d039-684f6b84bc-5xm54"] Nov 03 00:36:51.833665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.813893 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" event={"ID":"81b79a89-52ca-4440-9874-cc9a78bc0735","Type":"ContainerStarted","Data":"6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4"} Nov 03 00:36:51.997734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.997706 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="945471fa-324a-4df9-9d11-381caf2aa686" path="/var/lib/kubelet/pods/945471fa-324a-4df9-9d11-381caf2aa686/volumes" Nov 03 00:36:51.998138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:51.998124 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd3250bf-17b8-4836-9129-11f3809e35b6" path="/var/lib/kubelet/pods/dd3250bf-17b8-4836-9129-11f3809e35b6/volumes" Nov 03 00:36:52.512233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:52.512201 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:36:52.517656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:52.513624 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 00:36:52.517656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:52.513722 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 00:36:53.512525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:53.512483 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 00:36:53.513026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:53.512561 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 00:36:54.330657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:54.328711 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:36:54.512742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:54.512698 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 00:36:54.513134 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:54.512762 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 00:36:55.511833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:55.511797 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 00:36:55.511994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:55.511851 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 00:36:56.511800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:56.511764 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" start-of-body= Nov 03 00:36:56.512166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:56.511816 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.178:15021/healthz/ready\": dial tcp 10.132.0.178:15021: connect: connection refused" Nov 03 00:36:57.243512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.243478 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8"] Nov 03 00:36:57.244132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244118 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" Nov 03 00:36:57.244184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244134 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" Nov 03 00:36:57.244184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244146 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c3270557-ac9c-4c26-80cf-dcf9d00191f1" containerName="revoke-keys" Nov 03 00:36:57.244184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244151 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3270557-ac9c-4c26-80cf-dcf9d00191f1" containerName="revoke-keys" Nov 03 00:36:57.244184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244176 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dd3250bf-17b8-4836-9129-11f3809e35b6" containerName="payload-pre-processing" Nov 03 00:36:57.244184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244182 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd3250bf-17b8-4836-9129-11f3809e35b6" containerName="payload-pre-processing" Nov 03 00:36:57.244336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244197 5144 cpu_manager.go:414] "containerMap: removing container" podUID="945471fa-324a-4df9-9d11-381caf2aa686" containerName="payload-processing" Nov 03 00:36:57.244336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244202 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="945471fa-324a-4df9-9d11-381caf2aa686" containerName="payload-processing" Nov 03 00:36:57.244399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244359 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd3250bf-17b8-4836-9129-11f3809e35b6" containerName="payload-pre-processing" Nov 03 00:36:57.244399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244381 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="945471fa-324a-4df9-9d11-381caf2aa686" containerName="payload-processing" Nov 03 00:36:57.244399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244391 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1af72847-0d9a-4cc3-80b8-36689800df6d" containerName="istio-proxy" Nov 03 00:36:57.244399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.244397 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3270557-ac9c-4c26-80cf-dcf9d00191f1" containerName="revoke-keys" Nov 03 00:36:57.249274 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.249254 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.303518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.303484 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8"] Nov 03 00:36:57.353670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.350946 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-75dfee\"" Nov 03 00:36:57.353670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.351129 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-75dfee-dockercfg-ctz2p\"" Nov 03 00:36:57.437265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.437231 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.438169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.438127 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsggq\" (UniqueName: \"kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq\") pod \"payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.449515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.449486 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68"] Nov 03 00:36:57.455075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.455054 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.506238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.506212 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68"] Nov 03 00:36:57.518244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.518220 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:36:57.519598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.519571 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:36:57.539985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.539952 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.540093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.540036 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhvb\" (UniqueName: \"kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb\") pod \"payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.540224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.540204 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsggq\" (UniqueName: \"kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq\") pod \"payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.540360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.540342 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.540681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.540647 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.597410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.597379 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsggq\" (UniqueName: \"kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq\") pod \"payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:57.641809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.641769 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.641936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.641903 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhvb\" (UniqueName: \"kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb\") pod \"payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.642550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.642529 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.699984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.699950 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhvb\" (UniqueName: \"kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb\") pod \"payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.769851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.769778 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:57.866394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:57.865068 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:58.161267 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:58.161229 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8edc800a_6aee_4bdb_940d_83ee6312bbf1.slice/crio-7af21113b39d46f370c2454ba0d55f4ae6f3e89f9ef59044c3a4458d410e5cb8 WatchSource:0}: Error finding container 7af21113b39d46f370c2454ba0d55f4ae6f3e89f9ef59044c3a4458d410e5cb8: Status 404 returned error can't find the container with id 7af21113b39d46f370c2454ba0d55f4ae6f3e89f9ef59044c3a4458d410e5cb8 Nov 03 00:36:58.161267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.161236 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68"] Nov 03 00:36:58.268108 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:36:58.268077 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff98dc40_87e8_42c9_8b07_8d7f478520a8.slice/crio-9c13255cdc5db75d0c6892f34baf6aaa37c8ce460a386204de4d9ffbc547da68 WatchSource:0}: Error finding container 9c13255cdc5db75d0c6892f34baf6aaa37c8ce460a386204de4d9ffbc547da68: Status 404 returned error can't find the container with id 9c13255cdc5db75d0c6892f34baf6aaa37c8ce460a386204de4d9ffbc547da68 Nov 03 00:36:58.268222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.268159 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8"] Nov 03 00:36:58.921837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.921798 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" event={"ID":"ff98dc40-87e8-42c9-8b07-8d7f478520a8","Type":"ContainerStarted","Data":"cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673"} Nov 03 00:36:58.922282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.921866 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:36:58.922282 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.921892 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" event={"ID":"ff98dc40-87e8-42c9-8b07-8d7f478520a8","Type":"ContainerStarted","Data":"9c13255cdc5db75d0c6892f34baf6aaa37c8ce460a386204de4d9ffbc547da68"} Nov 03 00:36:58.923691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.923625 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" event={"ID":"8edc800a-6aee-4bdb-940d-83ee6312bbf1","Type":"ContainerStarted","Data":"de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99"} Nov 03 00:36:58.923816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.923717 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" event={"ID":"8edc800a-6aee-4bdb-940d-83ee6312bbf1","Type":"ContainerStarted","Data":"7af21113b39d46f370c2454ba0d55f4ae6f3e89f9ef59044c3a4458d410e5cb8"} Nov 03 00:36:58.923816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:58.923747 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:36:59.034943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:59.034893 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" podStartSLOduration=2.03488155 podStartE2EDuration="2.03488155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:59.033880759 +0000 UTC m=+1909.179575215" watchObservedRunningTime="2026-11-03 00:36:59.03488155 +0000 UTC m=+1909.180576061" Nov 03 00:36:59.149663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:36:59.147922 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" podStartSLOduration=2.147903285 podStartE2EDuration="2.147903285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:59.145589218 +0000 UTC m=+1909.291283677" watchObservedRunningTime="2026-11-03 00:36:59.147903285 +0000 UTC m=+1909.293597743" Nov 03 00:37:07.507798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:07.507764 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5"] Nov 03 00:37:07.508210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:07.508006 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" containerID="cri-o://031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a" gracePeriod=30 Nov 03 00:37:07.546448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:07.546417 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:37:07.546568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:07.546475 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:37:09.936453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:09.936426 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:37:09.936866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:09.936501 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:37:11.671674 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:11.671644 5144 scope.go:122] "RemoveContainer" containerID="e3bb2b4a89e009c91e0e2ef34c95096fce389a99b2b56c36a4879c4c69787dfd" Nov 03 00:37:12.160748 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:37:12.160712 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:37:12.785756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.785690 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:37:12.842053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842019 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842102 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842141 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842178 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842227 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842263 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842381 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842458 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:12.842672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842475 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842547 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data\") pod \"3f02589b-9510-4e1e-9535-42cda380ff19\" (UID: \"3f02589b-9510-4e1e-9535-42cda380ff19\") " Nov 03 00:37:12.842672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842577 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:12.842938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.842916 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:12.843150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.843127 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:12.843263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.843177 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:12.843263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.843234 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.843263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.843251 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3f02589b-9510-4e1e-9535-42cda380ff19-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.843590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.843265 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.845085 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.844966 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:37:12.845206 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.845133 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "kube-api-access-fqftk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:12.845312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.845278 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:12.845372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.845308 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token" pod "3f02589b-9510-4e1e-9535-42cda380ff19" (UID: "3f02589b-9510-4e1e-9535-42cda380ff19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:12.945244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.945202 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.945244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.945242 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.945449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.945258 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3f02589b-9510-4e1e-9535-42cda380ff19-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.945449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.945271 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.945449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.945285 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3f02589b-9510-4e1e-9535-42cda380ff19-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:12.945449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:12.945298 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqftk\" (UniqueName: \"kubernetes.io/projected/3f02589b-9510-4e1e-9535-42cda380ff19-kube-api-access-fqftk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:13.040180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.040088 5144 generic.go:356] "Generic (PLEG): container finished" podID="3f02589b-9510-4e1e-9535-42cda380ff19" containerID="031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a" exitCode=0 Nov 03 00:37:13.040180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.040146 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" Nov 03 00:37:13.040410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.040175 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" event={"ID":"3f02589b-9510-4e1e-9535-42cda380ff19","Type":"ContainerDied","Data":"031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a"} Nov 03 00:37:13.040410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.040239 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5" event={"ID":"3f02589b-9510-4e1e-9535-42cda380ff19","Type":"ContainerDied","Data":"ea67fb3db94533d21a4600e399dc177c1ba905a816ab22a4ae0e5203abf8c45b"} Nov 03 00:37:13.040410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.040269 5144 scope.go:122] "RemoveContainer" containerID="031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a" Nov 03 00:37:13.051722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.051706 5144 scope.go:122] "RemoveContainer" containerID="930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2" Nov 03 00:37:13.063525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.063505 5144 scope.go:122] "RemoveContainer" containerID="031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a" Nov 03 00:37:13.063825 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:37:13.063808 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a\": container with ID starting with 031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a not found: ID does not exist" containerID="031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a" Nov 03 00:37:13.063881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.063834 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a"} err="failed to get container status \"031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a\": rpc error: code = NotFound desc = could not find container \"031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a\": container with ID starting with 031cae9807606681282d101d2fa2404a8ccc6aa12b7cae0c918f3782eac41d4a not found: ID does not exist" Nov 03 00:37:13.063881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.063852 5144 scope.go:122] "RemoveContainer" containerID="930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2" Nov 03 00:37:13.064116 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:37:13.064102 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2\": container with ID starting with 930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2 not found: ID does not exist" containerID="930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2" Nov 03 00:37:13.064164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.064119 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2"} err="failed to get container status \"930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2\": rpc error: code = NotFound desc = could not find container \"930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2\": container with ID starting with 930e405066f969c420c7e256ebce50591307e55f6a5fc8ac4d8e0fc7c97357d2 not found: ID does not exist" Nov 03 00:37:13.205512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.205465 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5"] Nov 03 00:37:13.210316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.210287 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3df441-openshift-default-7f89955d67-5dpz5"] Nov 03 00:37:13.997036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:13.997004 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" path="/var/lib/kubelet/pods/3f02589b-9510-4e1e-9535-42cda380ff19/volumes" Nov 03 00:37:24.450545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.450464 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl"] Nov 03 00:37:24.451117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.451103 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" Nov 03 00:37:24.451168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.451118 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" Nov 03 00:37:24.451259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.451249 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" Nov 03 00:37:24.451294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.451266 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" Nov 03 00:37:24.456432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.456413 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.511123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.511074 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl"] Nov 03 00:37:24.563669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.561870 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-0db2fb\"" Nov 03 00:37:24.563669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.562052 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-0db2fb-dockercfg-nl5jr\"" Nov 03 00:37:24.582457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.582416 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.582703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.582568 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdqmd\" (UniqueName: \"kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd\") pod \"payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.613666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.613529 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt"] Nov 03 00:37:24.629260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.629215 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" Nov 03 00:37:24.629260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.629252 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f02589b-9510-4e1e-9535-42cda380ff19" containerName="istio-proxy" Nov 03 00:37:24.641046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.638086 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.664800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.664772 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt"] Nov 03 00:37:24.684021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.683990 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.684183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.684134 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdqmd\" (UniqueName: \"kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd\") pod \"payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.684183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.684164 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.684303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.684260 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trdjn\" (UniqueName: \"kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn\") pod \"payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.684864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.684844 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.742163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.742101 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdqmd\" (UniqueName: \"kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd\") pod \"payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.770572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.770546 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:24.789503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.789475 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.789829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.789805 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trdjn\" (UniqueName: \"kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn\") pod \"payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.791247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.791222 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.849896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.849288 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trdjn\" (UniqueName: \"kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn\") pod \"payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:24.955701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:24.955668 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:25.170906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.170874 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl"] Nov 03 00:37:25.171121 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:37:25.171093 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5b7f1fc_c31a_4c7e_abfe_398c93837403.slice/crio-582039eb69311fff0b1cd3253fdfd2afa59952b9e4aae68c44d000cc48225a33 WatchSource:0}: Error finding container 582039eb69311fff0b1cd3253fdfd2afa59952b9e4aae68c44d000cc48225a33: Status 404 returned error can't find the container with id 582039eb69311fff0b1cd3253fdfd2afa59952b9e4aae68c44d000cc48225a33 Nov 03 00:37:25.336095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.336069 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt"] Nov 03 00:37:25.336332 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:37:25.336309 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod387ac7a3_ad6b_4ea2_9ab6_5afd92ff93ac.slice/crio-158ae42abaa6ea5b1687163751eddb7c94ff1283f80f96bab22b88cf1a9a724f WatchSource:0}: Error finding container 158ae42abaa6ea5b1687163751eddb7c94ff1283f80f96bab22b88cf1a9a724f: Status 404 returned error can't find the container with id 158ae42abaa6ea5b1687163751eddb7c94ff1283f80f96bab22b88cf1a9a724f Nov 03 00:37:25.362318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.362294 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" event={"ID":"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac","Type":"ContainerStarted","Data":"158ae42abaa6ea5b1687163751eddb7c94ff1283f80f96bab22b88cf1a9a724f"} Nov 03 00:37:25.363880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.363856 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" event={"ID":"e5b7f1fc-c31a-4c7e-abfe-398c93837403","Type":"ContainerStarted","Data":"50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4"} Nov 03 00:37:25.363966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.363905 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" event={"ID":"e5b7f1fc-c31a-4c7e-abfe-398c93837403","Type":"ContainerStarted","Data":"582039eb69311fff0b1cd3253fdfd2afa59952b9e4aae68c44d000cc48225a33"} Nov 03 00:37:25.364087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.364072 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:25.479669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:25.479594 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" podStartSLOduration=1.4795765269999999 podStartE2EDuration="1.479576527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:25.475808223 +0000 UTC m=+1935.621502679" watchObservedRunningTime="2026-11-03 00:37:25.479576527 +0000 UTC m=+1935.625270985" Nov 03 00:37:26.372910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:26.372873 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" event={"ID":"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac","Type":"ContainerStarted","Data":"37daaf8621f5975ab560438160cc54395934ba338935fe5e076a8d1cf89b6b10"} Nov 03 00:37:26.373142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:26.373114 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:26.487903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:26.487859 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" podStartSLOduration=2.487846492 podStartE2EDuration="2.487846492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:26.485250454 +0000 UTC m=+1936.630944908" watchObservedRunningTime="2026-11-03 00:37:26.487846492 +0000 UTC m=+1936.633540966" Nov 03 00:37:36.376188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:36.376160 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:37:37.383599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:37.383567 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:37:39.116155 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:39.116124 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79"] Nov 03 00:37:39.116598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:39.116325 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" containerID="cri-o://6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4" gracePeriod=30 Nov 03 00:37:42.156905 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:37:42.156867 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:37:42.521316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:42.521283 5144 patch_prober.go:29] interesting pod/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:37:42.521476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:42.521338 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:37:44.400246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.400167 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:37:44.445891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.445853 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.445921 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.445963 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446046 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446094 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:44.446222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446137 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446176 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446193 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:44.446222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446220 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446291 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446377 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert\") pod \"81b79a89-52ca-4440-9874-cc9a78bc0735\" (UID: \"81b79a89-52ca-4440-9874-cc9a78bc0735\") " Nov 03 00:37:44.446599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.446551 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:44.447495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.447429 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.447495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.447456 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.447495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.447471 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.447895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.447861 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:44.448389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.448357 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:37:44.449036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.448982 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:37:44.449036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.449018 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:37:44.449183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.449044 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:44.449183 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.449075 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587" pod "81b79a89-52ca-4440-9874-cc9a78bc0735" (UID: "81b79a89-52ca-4440-9874-cc9a78bc0735"). InnerVolumeSpecName "kube-api-access-jq587". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:37:44.537930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.537895 5144 generic.go:356] "Generic (PLEG): container finished" podID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerID="6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4" exitCode=0 Nov 03 00:37:44.538140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.537947 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" Nov 03 00:37:44.538140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.537984 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" event={"ID":"81b79a89-52ca-4440-9874-cc9a78bc0735","Type":"ContainerDied","Data":"6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4"} Nov 03 00:37:44.538140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.538056 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79" event={"ID":"81b79a89-52ca-4440-9874-cc9a78bc0735","Type":"ContainerDied","Data":"43d65a91f20069630ce8e267ed00d9e146d9718dab3d2f22509325ad8c3b6628"} Nov 03 00:37:44.538140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.538084 5144 scope.go:122] "RemoveContainer" containerID="6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4" Nov 03 00:37:44.549264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.549240 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81b79a89-52ca-4440-9874-cc9a78bc0735-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.549368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.549269 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.549368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.549283 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.549368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.549298 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.549368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.549311 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81b79a89-52ca-4440-9874-cc9a78bc0735-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.549368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.549324 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq587\" (UniqueName: \"kubernetes.io/projected/81b79a89-52ca-4440-9874-cc9a78bc0735-kube-api-access-jq587\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:37:44.551076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.551056 5144 scope.go:122] "RemoveContainer" containerID="e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017" Nov 03 00:37:44.562095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.562076 5144 scope.go:122] "RemoveContainer" containerID="6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4" Nov 03 00:37:44.562389 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:37:44.562364 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4\": container with ID starting with 6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4 not found: ID does not exist" containerID="6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4" Nov 03 00:37:44.562442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.562403 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4"} err="failed to get container status \"6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4\": rpc error: code = NotFound desc = could not find container \"6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4\": container with ID starting with 6ee0ad42d497e719b55a480f0c1aa19c1969aeceb22d2c4073cd03c31a5b5bd4 not found: ID does not exist" Nov 03 00:37:44.562442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.562429 5144 scope.go:122] "RemoveContainer" containerID="e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017" Nov 03 00:37:44.562754 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:37:44.562726 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017\": container with ID starting with e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017 not found: ID does not exist" containerID="e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017" Nov 03 00:37:44.562812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.562766 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017"} err="failed to get container status \"e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017\": rpc error: code = NotFound desc = could not find container \"e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017\": container with ID starting with e6429bc8ffc93d0e990c3445b62948f22f0270c41c8750de1b5bd5c1c66eb017 not found: ID does not exist" Nov 03 00:37:44.684035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.683946 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79"] Nov 03 00:37:44.688819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:44.688789 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-f2d039-openshift-default-78597bcf69-xhp79"] Nov 03 00:37:45.996843 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:45.996805 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" path="/var/lib/kubelet/pods/81b79a89-52ca-4440-9874-cc9a78bc0735/volumes" Nov 03 00:37:50.760885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.760848 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb"] Nov 03 00:37:50.761817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.761799 5144 cpu_manager.go:414] "containerMap: removing container" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" Nov 03 00:37:50.761885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.761820 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" Nov 03 00:37:50.761885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.761849 5144 cpu_manager.go:414] "containerMap: removing container" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" Nov 03 00:37:50.761885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.761857 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" Nov 03 00:37:50.762040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.762029 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" Nov 03 00:37:50.762087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.762049 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b79a89-52ca-4440-9874-cc9a78bc0735" containerName="istio-proxy" Nov 03 00:37:50.767785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.767766 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:50.818342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.818303 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb"] Nov 03 00:37:50.833334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.833296 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7ndq\" (UniqueName: \"kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq\") pod \"payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:50.833491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.833425 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:50.863606 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.863560 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-d61eae\"" Nov 03 00:37:50.863797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.863695 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-d61eae-dockercfg-jzswc\"" Nov 03 00:37:50.917659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.917598 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q"] Nov 03 00:37:50.924549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.924512 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:50.938273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.938238 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7ndq\" (UniqueName: \"kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq\") pod \"payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:50.941645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.941599 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:50.943267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.943077 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:50.988787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:50.987650 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q"] Nov 03 00:37:51.047393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.047306 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.047550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.047481 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74bfx\" (UniqueName: \"kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx\") pod \"payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.049109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.049085 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7ndq\" (UniqueName: \"kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq\") pod \"payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:51.082172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.082144 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:51.149482 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.149271 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.150972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.149700 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74bfx\" (UniqueName: \"kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx\") pod \"payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.150972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.150925 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.207300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.207265 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74bfx\" (UniqueName: \"kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx\") pod \"payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.298967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.298879 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:51.466833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.466805 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb"] Nov 03 00:37:51.467119 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:37:51.467097 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01a4cf82_5705_4290_8e72_80cf9985e6a0.slice/crio-ad3f71b0c11b638d3bdcff5e658d36a3a79fa4c955f1f5e1ffe37f8bafb25e2b WatchSource:0}: Error finding container ad3f71b0c11b638d3bdcff5e658d36a3a79fa4c955f1f5e1ffe37f8bafb25e2b: Status 404 returned error can't find the container with id ad3f71b0c11b638d3bdcff5e658d36a3a79fa4c955f1f5e1ffe37f8bafb25e2b Nov 03 00:37:51.607034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.606944 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" event={"ID":"01a4cf82-5705-4290-8e72-80cf9985e6a0","Type":"ContainerStarted","Data":"7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b"} Nov 03 00:37:51.607034 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.606999 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" event={"ID":"01a4cf82-5705-4290-8e72-80cf9985e6a0","Type":"ContainerStarted","Data":"ad3f71b0c11b638d3bdcff5e658d36a3a79fa4c955f1f5e1ffe37f8bafb25e2b"} Nov 03 00:37:51.607217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.607036 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:37:51.690008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.689983 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q"] Nov 03 00:37:51.691367 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:37:51.691340 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5c23b68_630d_483f_9720_4070d6743874.slice/crio-4aaf03d0c57aac89ba616d9be237dc4c0373bf6a968709711c314fefca5bc010 WatchSource:0}: Error finding container 4aaf03d0c57aac89ba616d9be237dc4c0373bf6a968709711c314fefca5bc010: Status 404 returned error can't find the container with id 4aaf03d0c57aac89ba616d9be237dc4c0373bf6a968709711c314fefca5bc010 Nov 03 00:37:51.721856 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:51.721806 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" podStartSLOduration=1.721788451 podStartE2EDuration="1.721788451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:51.719285044 +0000 UTC m=+1961.864979521" watchObservedRunningTime="2026-11-03 00:37:51.721788451 +0000 UTC m=+1961.867482906" Nov 03 00:37:52.615445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:52.615406 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" event={"ID":"b5c23b68-630d-483f-9720-4070d6743874","Type":"ContainerStarted","Data":"b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654"} Nov 03 00:37:52.615874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:52.615472 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" event={"ID":"b5c23b68-630d-483f-9720-4070d6743874","Type":"ContainerStarted","Data":"4aaf03d0c57aac89ba616d9be237dc4c0373bf6a968709711c314fefca5bc010"} Nov 03 00:37:52.615874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:52.615655 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:37:52.731909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:37:52.731850 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" podStartSLOduration=2.731836867 podStartE2EDuration="2.731836867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:52.728058241 +0000 UTC m=+1962.873752698" watchObservedRunningTime="2026-11-03 00:37:52.731836867 +0000 UTC m=+1962.877531322" Nov 03 00:38:02.619079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:02.619048 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:38:03.627477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:03.627442 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:38:12.156493 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:38:12.156446 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:38:14.768038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.768008 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s"] Nov 03 00:38:14.774019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.773997 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.848523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.847558 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s"] Nov 03 00:38:14.871833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.871794 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.871858 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.871961 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.872009 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.872058 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872229 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.872214 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.872268 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.872343 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75bxv\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.872417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.872383 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.882280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.882254 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-ff10d6-openshift-default-dockercfg-7ds6m\"" Nov 03 00:38:14.973735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.973701 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.973961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.973758 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.973961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.973806 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75bxv\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.973961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.973839 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.973977 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974013 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974047 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974083 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974108 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974108 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974346 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.974558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974437 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.975040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.974750 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.975040 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.975001 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.976394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.976356 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:14.976570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:14.976549 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:15.031257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.031152 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:15.031477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.031310 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75bxv\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv\") pod \"e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:15.087777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.087746 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:15.533880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.528683 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s"] Nov 03 00:38:15.550847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.550812 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:15.550971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.550928 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:15.551039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.550973 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:38:15.805615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.805527 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" event={"ID":"a882e75d-edd8-459a-9c51-d2834907464a","Type":"ContainerStarted","Data":"f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204"} Nov 03 00:38:15.805615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.805589 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" event={"ID":"a882e75d-edd8-459a-9c51-d2834907464a","Type":"ContainerStarted","Data":"1dbfdf3b6156aab191580b2b77e259977688e8ad5c0134ef941eb126c84393e6"} Nov 03 00:38:15.926417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:15.926357 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podStartSLOduration=1.9263383219999999 podStartE2EDuration="1.926338322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:15.921523444 +0000 UTC m=+1986.067217913" watchObservedRunningTime="2026-11-03 00:38:15.926338322 +0000 UTC m=+1986.072032770" Nov 03 00:38:16.088087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:16.087995 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:16.093984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:16.093961 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:16.813164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:16.813130 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:16.814273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:16.814252 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:38:21.862184 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:21.862155 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vh6mt_308a18b8-bcee-45dc-b1e5-4b10fcabc3c1/payload-processing/0.log" Nov 03 00:38:22.117979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:22.117903 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vh6mt_308a18b8-bcee-45dc-b1e5-4b10fcabc3c1/payload-processing/0.log" Nov 03 00:38:22.138128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:22.138099 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7"] Nov 03 00:38:22.247128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:22.247088 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx"] Nov 03 00:38:26.666688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.666654 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5"] Nov 03 00:38:26.690654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.682300 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:26.727119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.727083 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5"] Nov 03 00:38:26.771621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.771585 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ff10d6\"" Nov 03 00:38:26.771854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.771691 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ff10d6-dockercfg-wdvpv\"" Nov 03 00:38:26.866511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.866474 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:26.866763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.866551 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgcg\" (UniqueName: \"kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg\") pod \"payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:26.967727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.967599 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgcg\" (UniqueName: \"kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg\") pod \"payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:26.967931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.967812 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:26.968447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:26.968427 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:27.025037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.025000 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgcg\" (UniqueName: \"kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg\") pod \"payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:27.139428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.139383 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" podUID="9ba1ead1-21a4-491b-8bf3-58a386204e13" containerName="payload-processing" containerID="cri-o://13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2" gracePeriod=25 Nov 03 00:38:27.247859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.247765 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" podUID="beee5189-ddaf-496f-9627-770f44f39571" containerName="payload-pre-processing" containerID="cri-o://e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3" gracePeriod=25 Nov 03 00:38:27.318992 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.318969 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:27.416458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.416090 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:38:27.576941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.576888 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume\") pod \"9ba1ead1-21a4-491b-8bf3-58a386204e13\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " Nov 03 00:38:27.577182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.577047 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49\" (UniqueName: \"kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49\") pod \"9ba1ead1-21a4-491b-8bf3-58a386204e13\" (UID: \"9ba1ead1-21a4-491b-8bf3-58a386204e13\") " Nov 03 00:38:27.578165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.578137 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume" pod "9ba1ead1-21a4-491b-8bf3-58a386204e13" (UID: "9ba1ead1-21a4-491b-8bf3-58a386204e13"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:27.582046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.582012 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49" pod "9ba1ead1-21a4-491b-8bf3-58a386204e13" (UID: "9ba1ead1-21a4-491b-8bf3-58a386204e13"). InnerVolumeSpecName "kube-api-access-rjm49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:27.591756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.591735 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:38:27.679152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.678974 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9ba1ead1-21a4-491b-8bf3-58a386204e13-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:27.679152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.679014 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjm49\" (UniqueName: \"kubernetes.io/projected/9ba1ead1-21a4-491b-8bf3-58a386204e13-kube-api-access-rjm49\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:27.732667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.732619 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5"] Nov 03 00:38:27.733257 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:38:27.733229 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf93604d2_5c44_405c_b07c_5dff7d3023dc.slice/crio-a1038b7154088e11e443f432718bffcc4ce063461355f1fff853efe3889eac01 WatchSource:0}: Error finding container a1038b7154088e11e443f432718bffcc4ce063461355f1fff853efe3889eac01: Status 404 returned error can't find the container with id a1038b7154088e11e443f432718bffcc4ce063461355f1fff853efe3889eac01 Nov 03 00:38:27.780049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.780017 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq\" (UniqueName: \"kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq\") pod \"beee5189-ddaf-496f-9627-770f44f39571\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " Nov 03 00:38:27.780185 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.780097 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume\") pod \"beee5189-ddaf-496f-9627-770f44f39571\" (UID: \"beee5189-ddaf-496f-9627-770f44f39571\") " Nov 03 00:38:27.780465 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.780439 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume" pod "beee5189-ddaf-496f-9627-770f44f39571" (UID: "beee5189-ddaf-496f-9627-770f44f39571"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:38:27.780762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.780746 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/beee5189-ddaf-496f-9627-770f44f39571-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:27.782428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.782402 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq" pod "beee5189-ddaf-496f-9627-770f44f39571" (UID: "beee5189-ddaf-496f-9627-770f44f39571"). InnerVolumeSpecName "kube-api-access-vwxtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:38:27.882197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.882103 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwxtq\" (UniqueName: \"kubernetes.io/projected/beee5189-ddaf-496f-9627-770f44f39571-kube-api-access-vwxtq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:38:27.907512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.907478 5144 generic.go:356] "Generic (PLEG): container finished" podID="9ba1ead1-21a4-491b-8bf3-58a386204e13" containerID="13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2" exitCode=0 Nov 03 00:38:27.907740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.907520 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" Nov 03 00:38:27.907740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.907563 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" event={"ID":"9ba1ead1-21a4-491b-8bf3-58a386204e13","Type":"ContainerDied","Data":"13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2"} Nov 03 00:38:27.907740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.907616 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7" event={"ID":"9ba1ead1-21a4-491b-8bf3-58a386204e13","Type":"ContainerDied","Data":"0ca8761ad4c3e52e36076edb0cafdd42d3a891e8c2dd17321b073d13028a594a"} Nov 03 00:38:27.907740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.907660 5144 scope.go:122] "RemoveContainer" containerID="13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2" Nov 03 00:38:27.909769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.909740 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" event={"ID":"f93604d2-5c44-405c-b07c-5dff7d3023dc","Type":"ContainerStarted","Data":"87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0"} Nov 03 00:38:27.909902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.909799 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" event={"ID":"f93604d2-5c44-405c-b07c-5dff7d3023dc","Type":"ContainerStarted","Data":"a1038b7154088e11e443f432718bffcc4ce063461355f1fff853efe3889eac01"} Nov 03 00:38:27.909902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.909845 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:27.911424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.911400 5144 generic.go:356] "Generic (PLEG): container finished" podID="beee5189-ddaf-496f-9627-770f44f39571" containerID="e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3" exitCode=0 Nov 03 00:38:27.911533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.911431 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" Nov 03 00:38:27.911533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.911497 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" event={"ID":"beee5189-ddaf-496f-9627-770f44f39571","Type":"ContainerDied","Data":"e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3"} Nov 03 00:38:27.911697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.911538 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx" event={"ID":"beee5189-ddaf-496f-9627-770f44f39571","Type":"ContainerDied","Data":"a91da3b2e18f0c5a26d8fbcfee18026e2ddd9948dad1712a21a1c5bbdea9da2d"} Nov 03 00:38:27.923280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.923248 5144 scope.go:122] "RemoveContainer" containerID="13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2" Nov 03 00:38:27.923733 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:38:27.923699 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2\": container with ID starting with 13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2 not found: ID does not exist" containerID="13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2" Nov 03 00:38:27.923847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.923738 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2"} err="failed to get container status \"13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2\": rpc error: code = NotFound desc = could not find container \"13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2\": container with ID starting with 13640685c8346a5241b062aa69dbdd9f9e14974b87767bcf4a73ef9d792383e2 not found: ID does not exist" Nov 03 00:38:27.923847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.923758 5144 scope.go:122] "RemoveContainer" containerID="e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3" Nov 03 00:38:27.934143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.934123 5144 scope.go:122] "RemoveContainer" containerID="e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3" Nov 03 00:38:27.934442 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:38:27.934418 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3\": container with ID starting with e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3 not found: ID does not exist" containerID="e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3" Nov 03 00:38:27.934514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:27.934452 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3"} err="failed to get container status \"e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3\": rpc error: code = NotFound desc = could not find container \"e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3\": container with ID starting with e641e03e0ac722c75bd57f3a5ea2b1f2e2cc10c484079e280678ada94f70e3e3 not found: ID does not exist" Nov 03 00:38:28.071721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:28.071688 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7"] Nov 03 00:38:28.077771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:28.077742 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-011885-85f8f7dd69-5d7b7"] Nov 03 00:38:28.187100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:28.186979 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" podStartSLOduration=2.186966174 podStartE2EDuration="2.186966174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:28.184507741 +0000 UTC m=+1998.330202216" watchObservedRunningTime="2026-11-03 00:38:28.186966174 +0000 UTC m=+1998.332660629" Nov 03 00:38:28.298294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:28.298225 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx"] Nov 03 00:38:28.313073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:28.313034 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-011885-6466c9999f-6tdzx"] Nov 03 00:38:29.996960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:29.996925 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ba1ead1-21a4-491b-8bf3-58a386204e13" path="/var/lib/kubelet/pods/9ba1ead1-21a4-491b-8bf3-58a386204e13/volumes" Nov 03 00:38:29.997343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:29.997334 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beee5189-ddaf-496f-9627-770f44f39571" path="/var/lib/kubelet/pods/beee5189-ddaf-496f-9627-770f44f39571/volumes" Nov 03 00:38:38.931815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:38:38.931782 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:38:42.159044 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:38:42.159004 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:39:12.156138 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:39:12.156042 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:39:31.970077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:31.970044 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s"] Nov 03 00:39:32.079131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:32.079074 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn"] Nov 03 00:39:36.971117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:36.971060 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" podUID="badebde3-1cd7-4860-a18c-3a4be1f02ed7" containerName="payload-processing" containerID="cri-o://5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007" gracePeriod=25 Nov 03 00:39:37.079533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.079495 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" podUID="1205ebf2-392b-46ee-a8a3-31d7f8b29739" containerName="payload-pre-processing" containerID="cri-o://bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1" gracePeriod=25 Nov 03 00:39:37.258553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.258520 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:39:37.361848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.361820 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj\" (UniqueName: \"kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj\") pod \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " Nov 03 00:39:37.362041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.362023 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume\") pod \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\" (UID: \"badebde3-1cd7-4860-a18c-3a4be1f02ed7\") " Nov 03 00:39:37.362452 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.362423 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume" pod "badebde3-1cd7-4860-a18c-3a4be1f02ed7" (UID: "badebde3-1cd7-4860-a18c-3a4be1f02ed7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:37.364376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.364352 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj" pod "badebde3-1cd7-4860-a18c-3a4be1f02ed7" (UID: "badebde3-1cd7-4860-a18c-3a4be1f02ed7"). InnerVolumeSpecName "kube-api-access-7sdnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:37.374552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.374525 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:39:37.463578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.463536 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sdnj\" (UniqueName: \"kubernetes.io/projected/badebde3-1cd7-4860-a18c-3a4be1f02ed7-kube-api-access-7sdnj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:37.463578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.463571 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/badebde3-1cd7-4860-a18c-3a4be1f02ed7-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:37.537399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.537351 5144 generic.go:356] "Generic (PLEG): container finished" podID="1205ebf2-392b-46ee-a8a3-31d7f8b29739" containerID="bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1" exitCode=0 Nov 03 00:39:37.537604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.537471 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" Nov 03 00:39:37.537604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.537462 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" event={"ID":"1205ebf2-392b-46ee-a8a3-31d7f8b29739","Type":"ContainerDied","Data":"bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1"} Nov 03 00:39:37.537745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.537662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn" event={"ID":"1205ebf2-392b-46ee-a8a3-31d7f8b29739","Type":"ContainerDied","Data":"bff3b8b0673de3e1e8968c02ac78f6abc0b5cf99d0916835ee04d7a314fc6fb9"} Nov 03 00:39:37.537745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.537696 5144 scope.go:122] "RemoveContainer" containerID="bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1" Nov 03 00:39:37.539729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.539685 5144 generic.go:356] "Generic (PLEG): container finished" podID="badebde3-1cd7-4860-a18c-3a4be1f02ed7" containerID="5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007" exitCode=0 Nov 03 00:39:37.539850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.539747 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" Nov 03 00:39:37.539916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.539890 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" event={"ID":"badebde3-1cd7-4860-a18c-3a4be1f02ed7","Type":"ContainerDied","Data":"5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007"} Nov 03 00:39:37.539963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.539931 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s" event={"ID":"badebde3-1cd7-4860-a18c-3a4be1f02ed7","Type":"ContainerDied","Data":"e455045778eb8c4f0710dffd5e2fbf4bfae22da511b71cc519b9c758df6b16f6"} Nov 03 00:39:37.551766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.551750 5144 scope.go:122] "RemoveContainer" containerID="bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1" Nov 03 00:39:37.552048 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:39:37.552031 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1\": container with ID starting with bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1 not found: ID does not exist" containerID="bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1" Nov 03 00:39:37.552112 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.552055 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1"} err="failed to get container status \"bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1\": rpc error: code = NotFound desc = could not find container \"bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1\": container with ID starting with bdbca6228c61af0c1357010189b5bcc2dc1ac2362ba40a98dc886d5e529c06a1 not found: ID does not exist" Nov 03 00:39:37.552112 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.552073 5144 scope.go:122] "RemoveContainer" containerID="5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007" Nov 03 00:39:37.563466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.563448 5144 scope.go:122] "RemoveContainer" containerID="5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007" Nov 03 00:39:37.563760 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:39:37.563737 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007\": container with ID starting with 5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007 not found: ID does not exist" containerID="5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007" Nov 03 00:39:37.563861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.563765 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007"} err="failed to get container status \"5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007\": rpc error: code = NotFound desc = could not find container \"5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007\": container with ID starting with 5ccd1996d37059c2fc028f5950e44d8b04612fe3e785a153ae94c54f9d19e007 not found: ID does not exist" Nov 03 00:39:37.564074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.564055 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume\") pod \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " Nov 03 00:39:37.564118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.564110 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k\" (UniqueName: \"kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k\") pod \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\" (UID: \"1205ebf2-392b-46ee-a8a3-31d7f8b29739\") " Nov 03 00:39:37.564424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.564404 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume" pod "1205ebf2-392b-46ee-a8a3-31d7f8b29739" (UID: "1205ebf2-392b-46ee-a8a3-31d7f8b29739"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:39:37.566293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.566270 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k" pod "1205ebf2-392b-46ee-a8a3-31d7f8b29739" (UID: "1205ebf2-392b-46ee-a8a3-31d7f8b29739"). InnerVolumeSpecName "kube-api-access-qct7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:39:37.665427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.665392 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1205ebf2-392b-46ee-a8a3-31d7f8b29739-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:37.665427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.665422 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qct7k\" (UniqueName: \"kubernetes.io/projected/1205ebf2-392b-46ee-a8a3-31d7f8b29739-kube-api-access-qct7k\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:39:37.704555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.704525 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s"] Nov 03 00:39:37.709864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.709839 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-cb486a-58886984c-69r2s"] Nov 03 00:39:37.958397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.958366 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn"] Nov 03 00:39:37.961719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.961694 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-cb486a-c7f874c6c-f9bqn"] Nov 03 00:39:37.997607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.997572 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1205ebf2-392b-46ee-a8a3-31d7f8b29739" path="/var/lib/kubelet/pods/1205ebf2-392b-46ee-a8a3-31d7f8b29739/volumes" Nov 03 00:39:37.998188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:37.998167 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="badebde3-1cd7-4860-a18c-3a4be1f02ed7" path="/var/lib/kubelet/pods/badebde3-1cd7-4860-a18c-3a4be1f02ed7/volumes" Nov 03 00:39:42.164507 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:39:42.164472 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:39:47.854862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:47.854812 5144 patch_prober.go:29] interesting pod/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:39:47.855365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:47.854891 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:39:48.628706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:48.628588 5144 generic.go:356] "Generic (PLEG): container finished" podID="a882e75d-edd8-459a-9c51-d2834907464a" containerID="f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204" exitCode=0 Nov 03 00:39:48.628706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:48.628663 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" event={"ID":"a882e75d-edd8-459a-9c51-d2834907464a","Type":"ContainerDied","Data":"f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204"} Nov 03 00:39:48.629131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:48.629114 5144 scope.go:122] "RemoveContainer" containerID="f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204" Nov 03 00:39:48.631022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:48.630993 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:39:48.631107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:48.631077 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:39:48.631143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:48.631118 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:39:49.651866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:49.651603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" event={"ID":"a882e75d-edd8-459a-9c51-d2834907464a","Type":"ContainerStarted","Data":"892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175"} Nov 03 00:39:50.087918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:50.087882 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:39:50.088118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:50.088063 5144 patch_prober.go:29] interesting pod/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 00:39:50.088200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:50.088119 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 00:39:51.088244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:51.088214 5144 patch_prober.go:29] interesting pod/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 00:39:51.088710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:51.088266 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 00:39:52.088540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:52.088512 5144 patch_prober.go:29] interesting pod/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 00:39:52.088988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:52.088557 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 00:39:53.088509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:53.088465 5144 patch_prober.go:29] interesting pod/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 00:39:53.088804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:53.088527 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 00:39:54.051232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:54.051201 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp"] Nov 03 00:39:54.088544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:54.088508 5144 patch_prober.go:29] interesting pod/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" start-of-body= Nov 03 00:39:54.088748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:54.088564 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.196:15021/healthz/ready\": dial tcp 10.132.0.196:15021: connect: connection refused" Nov 03 00:39:54.101925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:54.101891 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70b82998-49prp"] Nov 03 00:39:55.094325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:55.094297 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:39:55.094760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:55.094742 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:39:55.095383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:55.095368 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:39:55.996639 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:39:55.996594 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4361e5c6-f389-4d14-b71c-93e3bd82ba32" path="/var/lib/kubelet/pods/4361e5c6-f389-4d14-b71c-93e3bd82ba32/volumes" Nov 03 00:40:00.008751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.008717 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf"] Nov 03 00:40:00.009365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009346 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1205ebf2-392b-46ee-a8a3-31d7f8b29739" containerName="payload-pre-processing" Nov 03 00:40:00.009365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009365 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1205ebf2-392b-46ee-a8a3-31d7f8b29739" containerName="payload-pre-processing" Nov 03 00:40:00.009513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009397 5144 cpu_manager.go:414] "containerMap: removing container" podUID="beee5189-ddaf-496f-9627-770f44f39571" containerName="payload-pre-processing" Nov 03 00:40:00.009513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009402 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beee5189-ddaf-496f-9627-770f44f39571" containerName="payload-pre-processing" Nov 03 00:40:00.009513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009433 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9ba1ead1-21a4-491b-8bf3-58a386204e13" containerName="payload-processing" Nov 03 00:40:00.009513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009439 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ba1ead1-21a4-491b-8bf3-58a386204e13" containerName="payload-processing" Nov 03 00:40:00.009513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009451 5144 cpu_manager.go:414] "containerMap: removing container" podUID="badebde3-1cd7-4860-a18c-3a4be1f02ed7" containerName="payload-processing" Nov 03 00:40:00.009513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009456 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="badebde3-1cd7-4860-a18c-3a4be1f02ed7" containerName="payload-processing" Nov 03 00:40:00.009852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009556 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="beee5189-ddaf-496f-9627-770f44f39571" containerName="payload-pre-processing" Nov 03 00:40:00.009852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009567 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ba1ead1-21a4-491b-8bf3-58a386204e13" containerName="payload-processing" Nov 03 00:40:00.009852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009579 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="badebde3-1cd7-4860-a18c-3a4be1f02ed7" containerName="payload-processing" Nov 03 00:40:00.009852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.009588 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1205ebf2-392b-46ee-a8a3-31d7f8b29739" containerName="payload-pre-processing" Nov 03 00:40:00.013131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.013107 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.074046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.073947 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf"] Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.116926 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bdd194-openshift-default-dockercfg-nx7fh\"" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127061 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127115 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127169 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127225 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127256 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127318 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx2s6\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127357 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127401 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.133951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.127464 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.228974 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229044 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229108 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229178 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229298 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229400 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx2s6\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229436 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229487 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229563 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229572 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.229777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.229736 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.230117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.230089 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.230208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.230128 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.230378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.230355 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.231562 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.231541 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.231746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.231726 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.285289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.285224 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.285395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.285336 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx2s6\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6\") pod \"e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.327554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.327528 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:00.778946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.778917 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf"] Nov 03 00:40:00.779500 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:40:00.779463 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde5382a4_91d6_44f4_8911_dfac0b6d7382.slice/crio-409da360ca765181e0922fd2d5a843536219e654002517fe3ecac9063a04ed5d WatchSource:0}: Error finding container 409da360ca765181e0922fd2d5a843536219e654002517fe3ecac9063a04ed5d: Status 404 returned error can't find the container with id 409da360ca765181e0922fd2d5a843536219e654002517fe3ecac9063a04ed5d Nov 03 00:40:00.782240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.782207 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:40:00.782385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.782274 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:40:00.782385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:00.782302 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:40:01.742505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:01.742464 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" event={"ID":"de5382a4-91d6-44f4-8911-dfac0b6d7382","Type":"ContainerStarted","Data":"511837c54d27ba68078cceaea432913786c022cf048bd9858d7f55a345aebe44"} Nov 03 00:40:01.742981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:01.742526 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" event={"ID":"de5382a4-91d6-44f4-8911-dfac0b6d7382","Type":"ContainerStarted","Data":"409da360ca765181e0922fd2d5a843536219e654002517fe3ecac9063a04ed5d"} Nov 03 00:40:01.910299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:01.910252 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podStartSLOduration=2.9102387 podStartE2EDuration="2.9102387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:01.90707923 +0000 UTC m=+2092.052773713" watchObservedRunningTime="2026-11-03 00:40:01.9102387 +0000 UTC m=+2092.055933199" Nov 03 00:40:02.328602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:02.328573 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:02.334156 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:02.334134 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:02.749455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:02.749360 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:02.750320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:02.750300 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:40:10.503319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:10.503289 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:40:10.512354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:10.512295 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:40:12.155318 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:40:12.155242 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:40:19.615643 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:19.615558 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w_8c29fecd-d6b1-4d1d-b561-9bcac32b83a7/payload-processing/0.log" Nov 03 00:40:19.901284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:19.901201 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vh6mt_308a18b8-bcee-45dc-b1e5-4b10fcabc3c1/payload-processing/0.log" Nov 03 00:40:20.147544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:20.147515 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-vh6mt_308a18b8-bcee-45dc-b1e5-4b10fcabc3c1/payload-processing/0.log" Nov 03 00:40:27.869152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:27.869116 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt"] Nov 03 00:40:27.982412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:27.982382 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl"] Nov 03 00:40:32.870375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:32.870328 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" podUID="387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" containerName="payload-processing" containerID="cri-o://37daaf8621f5975ab560438160cc54395934ba338935fe5e076a8d1cf89b6b10" gracePeriod=25 Nov 03 00:40:32.983530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:32.983488 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" podUID="e5b7f1fc-c31a-4c7e-abfe-398c93837403" containerName="payload-pre-processing" containerID="cri-o://50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4" gracePeriod=25 Nov 03 00:40:33.019828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.019660 5144 generic.go:356] "Generic (PLEG): container finished" podID="387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" containerID="37daaf8621f5975ab560438160cc54395934ba338935fe5e076a8d1cf89b6b10" exitCode=0 Nov 03 00:40:33.019828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.019740 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" event={"ID":"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac","Type":"ContainerDied","Data":"37daaf8621f5975ab560438160cc54395934ba338935fe5e076a8d1cf89b6b10"} Nov 03 00:40:33.163553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.163528 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:40:33.287369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.287339 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:40:33.292394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.292364 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume\") pod \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " Nov 03 00:40:33.292752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.292729 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn\" (UniqueName: \"kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn\") pod \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\" (UID: \"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac\") " Nov 03 00:40:33.292826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.292792 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume" pod "387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" (UID: "387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:40:33.293235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.293217 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:33.296081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.296049 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn" pod "387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" (UID: "387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac"). InnerVolumeSpecName "kube-api-access-trdjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:33.393879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.393805 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume\") pod \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " Nov 03 00:40:33.394023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.394007 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd\" (UniqueName: \"kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd\") pod \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\" (UID: \"e5b7f1fc-c31a-4c7e-abfe-398c93837403\") " Nov 03 00:40:33.394148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.394127 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume" pod "e5b7f1fc-c31a-4c7e-abfe-398c93837403" (UID: "e5b7f1fc-c31a-4c7e-abfe-398c93837403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:40:33.394440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.394424 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trdjn\" (UniqueName: \"kubernetes.io/projected/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac-kube-api-access-trdjn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:33.394478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.394443 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5b7f1fc-c31a-4c7e-abfe-398c93837403-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:33.396147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.396126 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd" pod "e5b7f1fc-c31a-4c7e-abfe-398c93837403" (UID: "e5b7f1fc-c31a-4c7e-abfe-398c93837403"). InnerVolumeSpecName "kube-api-access-kdqmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:40:33.502690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:33.496328 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdqmd\" (UniqueName: \"kubernetes.io/projected/e5b7f1fc-c31a-4c7e-abfe-398c93837403-kube-api-access-kdqmd\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:40:34.028880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.028855 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" Nov 03 00:40:34.029295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.028880 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt" event={"ID":"387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac","Type":"ContainerDied","Data":"158ae42abaa6ea5b1687163751eddb7c94ff1283f80f96bab22b88cf1a9a724f"} Nov 03 00:40:34.029295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.028936 5144 scope.go:122] "RemoveContainer" containerID="37daaf8621f5975ab560438160cc54395934ba338935fe5e076a8d1cf89b6b10" Nov 03 00:40:34.031551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.031521 5144 generic.go:356] "Generic (PLEG): container finished" podID="e5b7f1fc-c31a-4c7e-abfe-398c93837403" containerID="50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4" exitCode=0 Nov 03 00:40:34.031720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.031589 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" Nov 03 00:40:34.031720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.031600 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" event={"ID":"e5b7f1fc-c31a-4c7e-abfe-398c93837403","Type":"ContainerDied","Data":"50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4"} Nov 03 00:40:34.031720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.031678 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl" event={"ID":"e5b7f1fc-c31a-4c7e-abfe-398c93837403","Type":"ContainerDied","Data":"582039eb69311fff0b1cd3253fdfd2afa59952b9e4aae68c44d000cc48225a33"} Nov 03 00:40:34.040670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.040612 5144 scope.go:122] "RemoveContainer" containerID="50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4" Nov 03 00:40:34.052925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.052905 5144 scope.go:122] "RemoveContainer" containerID="50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4" Nov 03 00:40:34.053279 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:40:34.053254 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4\": container with ID starting with 50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4 not found: ID does not exist" containerID="50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4" Nov 03 00:40:34.053376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.053288 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4"} err="failed to get container status \"50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4\": rpc error: code = NotFound desc = could not find container \"50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4\": container with ID starting with 50391387d63679391c2cb5ded83e07297624cc8c40af84cb10c824e3820ebbe4 not found: ID does not exist" Nov 03 00:40:34.193230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.193166 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl"] Nov 03 00:40:34.196524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.196501 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-0db2fb-6d4fc994cf-9lckl"] Nov 03 00:40:34.355793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.355758 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt"] Nov 03 00:40:34.360228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:34.360197 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-0db2fb-6cc94bbfdf-xdgdt"] Nov 03 00:40:35.997995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:35.997965 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" path="/var/lib/kubelet/pods/387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac/volumes" Nov 03 00:40:35.998446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:40:35.998430 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5b7f1fc-c31a-4c7e-abfe-398c93837403" path="/var/lib/kubelet/pods/e5b7f1fc-c31a-4c7e-abfe-398c93837403/volumes" Nov 03 00:40:42.162672 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:40:42.162621 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:41:03.543555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.543523 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh"] Nov 03 00:41:03.544186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.544171 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e5b7f1fc-c31a-4c7e-abfe-398c93837403" containerName="payload-pre-processing" Nov 03 00:41:03.544234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.544188 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5b7f1fc-c31a-4c7e-abfe-398c93837403" containerName="payload-pre-processing" Nov 03 00:41:03.544234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.544228 5144 cpu_manager.go:414] "containerMap: removing container" podUID="387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" containerName="payload-processing" Nov 03 00:41:03.544234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.544234 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" containerName="payload-processing" Nov 03 00:41:03.544362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.544353 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5b7f1fc-c31a-4c7e-abfe-398c93837403" containerName="payload-pre-processing" Nov 03 00:41:03.544402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.544369 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="387ac7a3-ad6b-4ea2-9ab6-5afd92ff93ac" containerName="payload-processing" Nov 03 00:41:03.547908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.547892 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.604771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.604743 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh"] Nov 03 00:41:03.656704 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.656670 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:41:03.658186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.657236 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:41:03.698547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.698510 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.698747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.698618 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.698796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.698758 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zpf6\" (UniqueName: \"kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.799853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.799760 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.799853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.799819 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.800100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.799882 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zpf6\" (UniqueName: \"kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.800515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.800495 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.802277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.802254 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.858805 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.858773 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zpf6\" (UniqueName: \"kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6\") pod \"maas-api-revoke-sub-859ed6fe-gldvh\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:03.862534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:03.862512 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:04.244005 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:41:04.243971 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97f0b73c_20cc_45bd_8bdc_cce59b1619f1.slice/crio-55302c25b2c1aab6943a09f0648771f2485d34f2b4affa848cc44a0ad0995504 WatchSource:0}: Error finding container 55302c25b2c1aab6943a09f0648771f2485d34f2b4affa848cc44a0ad0995504: Status 404 returned error can't find the container with id 55302c25b2c1aab6943a09f0648771f2485d34f2b4affa848cc44a0ad0995504 Nov 03 00:41:04.244970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:04.244945 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh"] Nov 03 00:41:04.294415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:04.294378 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" event={"ID":"97f0b73c-20cc-45bd-8bdc-cce59b1619f1","Type":"ContainerStarted","Data":"55302c25b2c1aab6943a09f0648771f2485d34f2b4affa848cc44a0ad0995504"} Nov 03 00:41:05.303254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:05.303227 5144 generic.go:356] "Generic (PLEG): container finished" podID="97f0b73c-20cc-45bd-8bdc-cce59b1619f1" containerID="ad994fd18c57538bb4616485f8960d7a64440fa7543851d132a9f41391ee6e8d" exitCode=0 Nov 03 00:41:05.303587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:05.303310 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" event={"ID":"97f0b73c-20cc-45bd-8bdc-cce59b1619f1","Type":"ContainerDied","Data":"ad994fd18c57538bb4616485f8960d7a64440fa7543851d132a9f41391ee6e8d"} Nov 03 00:41:06.480445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.480421 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:06.530357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.530322 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca\") pod \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " Nov 03 00:41:06.530504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.530390 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca\") pod \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " Nov 03 00:41:06.530576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.530547 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6\" (UniqueName: \"kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6\") pod \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\" (UID: \"97f0b73c-20cc-45bd-8bdc-cce59b1619f1\") " Nov 03 00:41:06.530757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.530731 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca" pod "97f0b73c-20cc-45bd-8bdc-cce59b1619f1" (UID: "97f0b73c-20cc-45bd-8bdc-cce59b1619f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:06.531129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.531034 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:06.532681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.532662 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca" pod "97f0b73c-20cc-45bd-8bdc-cce59b1619f1" (UID: "97f0b73c-20cc-45bd-8bdc-cce59b1619f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:41:06.532765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.532676 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6" pod "97f0b73c-20cc-45bd-8bdc-cce59b1619f1" (UID: "97f0b73c-20cc-45bd-8bdc-cce59b1619f1"). InnerVolumeSpecName "kube-api-access-4zpf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:06.632670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.632579 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:06.632670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:06.632608 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zpf6\" (UniqueName: \"kubernetes.io/projected/97f0b73c-20cc-45bd-8bdc-cce59b1619f1-kube-api-access-4zpf6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:07.329748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:07.329719 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" Nov 03 00:41:07.329949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:07.329754 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh" event={"ID":"97f0b73c-20cc-45bd-8bdc-cce59b1619f1","Type":"ContainerDied","Data":"55302c25b2c1aab6943a09f0648771f2485d34f2b4affa848cc44a0ad0995504"} Nov 03 00:41:07.329949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:07.329819 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55302c25b2c1aab6943a09f0648771f2485d34f2b4affa848cc44a0ad0995504" Nov 03 00:41:11.860725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:11.860691 5144 scope.go:122] "RemoveContainer" containerID="af32dcdc7dcbfcca3aa5101201f4c3ef18145ae244db8073337f094b76eb17ea" Nov 03 00:41:11.881131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:11.881108 5144 scope.go:122] "RemoveContainer" containerID="80801308821cd1ade113ed372f3b6ab6cf75a7576d391cf14f4403bd80ca7e6c" Nov 03 00:41:12.156189 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:12.156027 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:41:18.690729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:18.690695 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w"] Nov 03 00:41:18.691180 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:18.690938 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" podUID="8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" containerName="payload-processing" containerID="cri-o://1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138" gracePeriod=30 Nov 03 00:41:18.751862 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:18.751834 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv"] Nov 03 00:41:18.752054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:18.752033 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" podUID="06f627b6-06f4-4745-97f1-32ffeda2a032" containerName="payload-pre-processing" containerID="cri-o://f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66" gracePeriod=30 Nov 03 00:41:19.073153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.073119 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:41:19.087893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.087870 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:41:19.198213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.198136 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x\" (UniqueName: \"kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x\") pod \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " Nov 03 00:41:19.198358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.198243 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662\" (UniqueName: \"kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662\") pod \"06f627b6-06f4-4745-97f1-32ffeda2a032\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " Nov 03 00:41:19.198358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.198276 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume\") pod \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\" (UID: \"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7\") " Nov 03 00:41:19.198447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.198414 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume\") pod \"06f627b6-06f4-4745-97f1-32ffeda2a032\" (UID: \"06f627b6-06f4-4745-97f1-32ffeda2a032\") " Nov 03 00:41:19.198753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.198724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume" pod "8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" (UID: "8c29fecd-d6b1-4d1d-b561-9bcac32b83a7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:19.198886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.198803 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume" pod "06f627b6-06f4-4745-97f1-32ffeda2a032" (UID: "06f627b6-06f4-4745-97f1-32ffeda2a032"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:19.201028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.201007 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662" pod "06f627b6-06f4-4745-97f1-32ffeda2a032" (UID: "06f627b6-06f4-4745-97f1-32ffeda2a032"). InnerVolumeSpecName "kube-api-access-wm662". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:19.201124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.201024 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x" pod "8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" (UID: "8c29fecd-d6b1-4d1d-b561-9bcac32b83a7"). InnerVolumeSpecName "kube-api-access-ltb9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:19.313666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.306612 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06f627b6-06f4-4745-97f1-32ffeda2a032-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:19.313666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.306681 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltb9x\" (UniqueName: \"kubernetes.io/projected/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-kube-api-access-ltb9x\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:19.359656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.356778 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm662\" (UniqueName: \"kubernetes.io/projected/06f627b6-06f4-4745-97f1-32ffeda2a032-kube-api-access-wm662\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:19.359656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.356822 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:19.447899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.447861 5144 generic.go:356] "Generic (PLEG): container finished" podID="8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" containerID="1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138" exitCode=0 Nov 03 00:41:19.448094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.447919 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" Nov 03 00:41:19.448094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.447936 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" event={"ID":"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7","Type":"ContainerDied","Data":"1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138"} Nov 03 00:41:19.448094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.447995 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w" event={"ID":"8c29fecd-d6b1-4d1d-b561-9bcac32b83a7","Type":"ContainerDied","Data":"dccef021162cd977c635d519d2c30042c58a0d91a4f0679d56b68537f0c39aee"} Nov 03 00:41:19.448094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.448013 5144 scope.go:122] "RemoveContainer" containerID="1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138" Nov 03 00:41:19.450000 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.449974 5144 generic.go:356] "Generic (PLEG): container finished" podID="06f627b6-06f4-4745-97f1-32ffeda2a032" containerID="f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66" exitCode=0 Nov 03 00:41:19.450226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.450207 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" event={"ID":"06f627b6-06f4-4745-97f1-32ffeda2a032","Type":"ContainerDied","Data":"f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66"} Nov 03 00:41:19.450275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.450261 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" event={"ID":"06f627b6-06f4-4745-97f1-32ffeda2a032","Type":"ContainerDied","Data":"92187c28f5306661bb4c3a29626b86883d95e3f1415ddf2ba01a239c9c9fa953"} Nov 03 00:41:19.450350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.450337 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv" Nov 03 00:41:19.460074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.460051 5144 scope.go:122] "RemoveContainer" containerID="1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138" Nov 03 00:41:19.460367 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:19.460342 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138\": container with ID starting with 1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138 not found: ID does not exist" containerID="1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138" Nov 03 00:41:19.460448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.460377 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138"} err="failed to get container status \"1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138\": rpc error: code = NotFound desc = could not find container \"1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138\": container with ID starting with 1cd15d1a81e98b132b64ab608f6daf42d6f7e56158be6b6c018f38a1a7026138 not found: ID does not exist" Nov 03 00:41:19.460448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.460399 5144 scope.go:122] "RemoveContainer" containerID="f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66" Nov 03 00:41:19.471288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.471272 5144 scope.go:122] "RemoveContainer" containerID="f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66" Nov 03 00:41:19.471547 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:19.471530 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66\": container with ID starting with f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66 not found: ID does not exist" containerID="f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66" Nov 03 00:41:19.471609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.471555 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66"} err="failed to get container status \"f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66\": rpc error: code = NotFound desc = could not find container \"f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66\": container with ID starting with f2d280c89e5218e2a4d113432726e32a18ad3702dd3f1d9c6856a279fa944f66 not found: ID does not exist" Nov 03 00:41:19.611737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.611699 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv"] Nov 03 00:41:19.616610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.616568 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1a883c-6cbb5fb98c-dglkv"] Nov 03 00:41:19.726128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.726089 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w"] Nov 03 00:41:19.731401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.731376 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1a883c-564449c458-gxf7w"] Nov 03 00:41:19.996664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.996585 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f627b6-06f4-4745-97f1-32ffeda2a032" path="/var/lib/kubelet/pods/06f627b6-06f4-4745-97f1-32ffeda2a032/volumes" Nov 03 00:41:19.997057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:19.997042 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" path="/var/lib/kubelet/pods/8c29fecd-d6b1-4d1d-b561-9bcac32b83a7/volumes" Nov 03 00:41:24.027259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:24.027227 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h"] Nov 03 00:41:24.027729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:24.027411 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" containerID="cri-o://006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b" gracePeriod=30 Nov 03 00:41:24.629237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:24.629196 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68"] Nov 03 00:41:24.661195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:24.661164 5144 patch_prober.go:29] interesting pod/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:41:24.661350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:24.661230 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:41:24.740391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:24.740352 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8"] Nov 03 00:41:29.297096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.297038 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:41:29.377097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377065 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377123 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377168 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377247 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377276 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377319 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377391 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377431 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:29.377568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377470 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377471 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:29.377723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377599 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo\") pod \"1d75a7a8-7808-4e76-94b1-be98213ea910\" (UID: \"1d75a7a8-7808-4e76-94b1-be98213ea910\") " Nov 03 00:41:29.377723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377578 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:29.377723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377606 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:29.377907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.377813 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:29.378113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.378095 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.378193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.378119 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d75a7a8-7808-4e76-94b1-be98213ea910-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.378193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.378134 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.378193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.378149 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.378193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.378163 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.379736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.379707 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:29.379962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.379938 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:41:29.380164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.380141 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "kube-api-access-9s8g2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:29.380398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.380379 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy" pod "1d75a7a8-7808-4e76-94b1-be98213ea910" (UID: "1d75a7a8-7808-4e76-94b1-be98213ea910"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:29.479379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.479344 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.479379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.479379 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8g2\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-kube-api-access-9s8g2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.479610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.479394 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.479610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.479407 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d75a7a8-7808-4e76-94b1-be98213ea910-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.524911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.524885 5144 generic.go:356] "Generic (PLEG): container finished" podID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerID="006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b" exitCode=0 Nov 03 00:41:29.525053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.524945 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" Nov 03 00:41:29.525053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.524948 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" event={"ID":"1d75a7a8-7808-4e76-94b1-be98213ea910","Type":"ContainerDied","Data":"006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b"} Nov 03 00:41:29.525140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.525064 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h" event={"ID":"1d75a7a8-7808-4e76-94b1-be98213ea910","Type":"ContainerDied","Data":"d3e772108455c1339c89923e657605f847e54bb5177065023a89ccb744dba140"} Nov 03 00:41:29.525140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.525086 5144 scope.go:122] "RemoveContainer" containerID="006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b" Nov 03 00:41:29.539208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.539188 5144 scope.go:122] "RemoveContainer" containerID="eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3" Nov 03 00:41:29.549557 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.549535 5144 scope.go:122] "RemoveContainer" containerID="006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b" Nov 03 00:41:29.549853 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:29.549832 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b\": container with ID starting with 006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b not found: ID does not exist" containerID="006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b" Nov 03 00:41:29.549919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.549860 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b"} err="failed to get container status \"006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b\": rpc error: code = NotFound desc = could not find container \"006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b\": container with ID starting with 006ab457b65c4d93435e5ab49b2eee0ffc30214e7f3df0de127b1fb58f55d28b not found: ID does not exist" Nov 03 00:41:29.549919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.549877 5144 scope.go:122] "RemoveContainer" containerID="eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3" Nov 03 00:41:29.550128 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:29.550107 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3\": container with ID starting with eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3 not found: ID does not exist" containerID="eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3" Nov 03 00:41:29.550178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.550135 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3"} err="failed to get container status \"eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3\": rpc error: code = NotFound desc = could not find container \"eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3\": container with ID starting with eb392ff7f5735f613c0988b5884450b591cecedadd282844b52e49a324ec97a3 not found: ID does not exist" Nov 03 00:41:29.629760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.629732 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" podUID="8edc800a-6aee-4bdb-940d-83ee6312bbf1" containerName="payload-processing" containerID="cri-o://de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99" gracePeriod=25 Nov 03 00:41:29.691888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.691833 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h"] Nov 03 00:41:29.695232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.695209 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1a883c-openshift-default-59c78957f9-kfk6h"] Nov 03 00:41:29.740962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.740914 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" podUID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" containerName="payload-pre-processing" containerID="cri-o://cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673" gracePeriod=25 Nov 03 00:41:29.905067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.905039 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:41:29.985769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.985742 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb\" (UniqueName: \"kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb\") pod \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " Nov 03 00:41:29.985910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.985899 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume\") pod \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\" (UID: \"8edc800a-6aee-4bdb-940d-83ee6312bbf1\") " Nov 03 00:41:29.986258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.986231 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume" pod "8edc800a-6aee-4bdb-940d-83ee6312bbf1" (UID: "8edc800a-6aee-4bdb-940d-83ee6312bbf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:29.986486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.986470 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8edc800a-6aee-4bdb-940d-83ee6312bbf1-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:29.988105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.988077 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb" pod "8edc800a-6aee-4bdb-940d-83ee6312bbf1" (UID: "8edc800a-6aee-4bdb-940d-83ee6312bbf1"). InnerVolumeSpecName "kube-api-access-wbhvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:29.998982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:29.998955 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" path="/var/lib/kubelet/pods/1d75a7a8-7808-4e76-94b1-be98213ea910/volumes" Nov 03 00:41:30.020115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.020097 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:41:30.087699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.087609 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume\") pod \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " Nov 03 00:41:30.087699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.087670 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq\" (UniqueName: \"kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq\") pod \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\" (UID: \"ff98dc40-87e8-42c9-8b07-8d7f478520a8\") " Nov 03 00:41:30.087984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.087962 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume" pod "ff98dc40-87e8-42c9-8b07-8d7f478520a8" (UID: "ff98dc40-87e8-42c9-8b07-8d7f478520a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:30.088291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.088271 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff98dc40-87e8-42c9-8b07-8d7f478520a8-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:30.088402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.088294 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbhvb\" (UniqueName: \"kubernetes.io/projected/8edc800a-6aee-4bdb-940d-83ee6312bbf1-kube-api-access-wbhvb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:30.090168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.090143 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq" pod "ff98dc40-87e8-42c9-8b07-8d7f478520a8" (UID: "ff98dc40-87e8-42c9-8b07-8d7f478520a8"). InnerVolumeSpecName "kube-api-access-hsggq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:30.189980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.189946 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsggq\" (UniqueName: \"kubernetes.io/projected/ff98dc40-87e8-42c9-8b07-8d7f478520a8-kube-api-access-hsggq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:30.535010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.534974 5144 generic.go:356] "Generic (PLEG): container finished" podID="8edc800a-6aee-4bdb-940d-83ee6312bbf1" containerID="de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99" exitCode=0 Nov 03 00:41:30.535010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.535014 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" Nov 03 00:41:30.535558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.535051 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" event={"ID":"8edc800a-6aee-4bdb-940d-83ee6312bbf1","Type":"ContainerDied","Data":"de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99"} Nov 03 00:41:30.535558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.535099 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68" event={"ID":"8edc800a-6aee-4bdb-940d-83ee6312bbf1","Type":"ContainerDied","Data":"7af21113b39d46f370c2454ba0d55f4ae6f3e89f9ef59044c3a4458d410e5cb8"} Nov 03 00:41:30.535558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.535128 5144 scope.go:122] "RemoveContainer" containerID="de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99" Nov 03 00:41:30.537114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.537016 5144 generic.go:356] "Generic (PLEG): container finished" podID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" containerID="cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673" exitCode=0 Nov 03 00:41:30.537114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.537070 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" Nov 03 00:41:30.537298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.537066 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" event={"ID":"ff98dc40-87e8-42c9-8b07-8d7f478520a8","Type":"ContainerDied","Data":"cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673"} Nov 03 00:41:30.537298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.537240 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" event={"ID":"ff98dc40-87e8-42c9-8b07-8d7f478520a8","Type":"ContainerDied","Data":"9c13255cdc5db75d0c6892f34baf6aaa37c8ce460a386204de4d9ffbc547da68"} Nov 03 00:41:30.547067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.547048 5144 scope.go:122] "RemoveContainer" containerID="de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99" Nov 03 00:41:30.547385 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:30.547363 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99\": container with ID starting with de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99 not found: ID does not exist" containerID="de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99" Nov 03 00:41:30.547494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.547395 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99"} err="failed to get container status \"de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99\": rpc error: code = NotFound desc = could not find container \"de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99\": container with ID starting with de65724f180810d1e619cc7a61e6f4af9ca83eed1c83367161be45386b729e99 not found: ID does not exist" Nov 03 00:41:30.547494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.547414 5144 scope.go:122] "RemoveContainer" containerID="cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673" Nov 03 00:41:30.558695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.558677 5144 scope.go:122] "RemoveContainer" containerID="cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673" Nov 03 00:41:30.558986 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:30.558969 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673\": container with ID starting with cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673 not found: ID does not exist" containerID="cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673" Nov 03 00:41:30.559032 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.558992 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673"} err="failed to get container status \"cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673\": rpc error: code = NotFound desc = could not find container \"cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673\": container with ID starting with cc4ea1ae85640962c94c36364afd71f4becbc9648ddd528bad712f902e96e673 not found: ID does not exist" Nov 03 00:41:30.701278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.701252 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68"] Nov 03 00:41:30.704614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.704592 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-75dfee-688d456b6c-zqk68"] Nov 03 00:41:30.812394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.812332 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8"] Nov 03 00:41:30.815316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.815297 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8"] Nov 03 00:41:30.933487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:30.933461 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-75dfee-6f545bb955-69lg8" podUID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.190:50052\" within 1s: context deadline exceeded" Nov 03 00:41:30.933618 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:41:30.933585 5144 logging.go:55] [core] [Channel #6338 SubChannel #6339]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.190:50052", ServerName: "10.132.0.190:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.190:50052: operation was canceled" Nov 03 00:41:32.000296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:32.000256 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8edc800a-6aee-4bdb-940d-83ee6312bbf1" path="/var/lib/kubelet/pods/8edc800a-6aee-4bdb-940d-83ee6312bbf1/volumes" Nov 03 00:41:32.001023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:32.001004 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" path="/var/lib/kubelet/pods/ff98dc40-87e8-42c9-8b07-8d7f478520a8/volumes" Nov 03 00:41:35.047476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:35.047397 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w"] Nov 03 00:41:35.085946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:35.085916 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ba0573f-gbc6w"] Nov 03 00:41:35.997373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:35.997340 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3270557-ac9c-4c26-80cf-dcf9d00191f1" path="/var/lib/kubelet/pods/c3270557-ac9c-4c26-80cf-dcf9d00191f1/volumes" Nov 03 00:41:42.165401 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:41:42.165361 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:41:46.717737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:46.717692 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k"] Nov 03 00:41:46.718251 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:46.717978 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" containerID="cri-o://eb38b2911e2feb3bafbcf813c7a628f529c2a2606b5898e339435e00ee2b648c" gracePeriod=30 Nov 03 00:41:48.558293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.558250 5144 patch_prober.go:29] interesting pod/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": read tcp 10.132.0.2:50560->10.132.0.198:15021: read: connection reset by peer" start-of-body= Nov 03 00:41:48.558733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.558320 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": read tcp 10.132.0.2:50560->10.132.0.198:15021: read: connection reset by peer" Nov 03 00:41:48.815223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.815142 5144 generic.go:356] "Generic (PLEG): container finished" podID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerID="511837c54d27ba68078cceaea432913786c022cf048bd9858d7f55a345aebe44" exitCode=0 Nov 03 00:41:48.815223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.815211 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" event={"ID":"de5382a4-91d6-44f4-8911-dfac0b6d7382","Type":"ContainerDied","Data":"511837c54d27ba68078cceaea432913786c022cf048bd9858d7f55a345aebe44"} Nov 03 00:41:48.815662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.815621 5144 scope.go:122] "RemoveContainer" containerID="511837c54d27ba68078cceaea432913786c022cf048bd9858d7f55a345aebe44" Nov 03 00:41:48.817149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.817122 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:41:48.817238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.817190 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:41:48.817238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:48.817220 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:41:49.849444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:49.849405 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" event={"ID":"de5382a4-91d6-44f4-8911-dfac0b6d7382","Type":"ContainerStarted","Data":"1218918e642ed21cf1a77ac981e06a82506e3591a6119c485a56326ef3c20d96"} Nov 03 00:41:50.330081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:50.330044 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:41:50.330511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:50.330387 5144 patch_prober.go:29] interesting pod/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 00:41:50.330785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:50.330759 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 00:41:51.329026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:51.328990 5144 patch_prober.go:29] interesting pod/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 00:41:51.329702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:51.329584 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 00:41:51.875757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:51.875721 5144 generic.go:356] "Generic (PLEG): container finished" podID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerID="eb38b2911e2feb3bafbcf813c7a628f529c2a2606b5898e339435e00ee2b648c" exitCode=0 Nov 03 00:41:51.875954 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:51.875826 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" event={"ID":"b3499f53-e055-4cc9-96a5-8a28e18dc3b9","Type":"ContainerDied","Data":"eb38b2911e2feb3bafbcf813c7a628f529c2a2606b5898e339435e00ee2b648c"} Nov 03 00:41:51.893675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:51.893619 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 00:41:51.893820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:51.893698 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 00:41:52.082626 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.082600 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:41:52.105423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105395 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105488 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105531 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105571 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105603 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105671 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105778 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105843 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105821 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.105905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105876 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:52.105905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105886 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:52.106070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105945 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy\") pod \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\" (UID: \"b3499f53-e055-4cc9-96a5-8a28e18dc3b9\") " Nov 03 00:41:52.106127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.105787 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:41:52.106173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106074 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:52.106173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106096 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:52.106771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106746 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.106885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106777 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.106885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106794 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.106885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106806 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.106885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.106818 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.107950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.107922 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:52.108059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.108032 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "kube-api-access-85vgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:41:52.108193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.108176 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:41:52.108370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.108352 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo" pod "b3499f53-e055-4cc9-96a5-8a28e18dc3b9" (UID: "b3499f53-e055-4cc9-96a5-8a28e18dc3b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:41:52.208334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.208302 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.208334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.208331 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.208334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.208340 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.208570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.208349 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85vgh\" (UniqueName: \"kubernetes.io/projected/b3499f53-e055-4cc9-96a5-8a28e18dc3b9-kube-api-access-85vgh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:41:52.335713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.334002 5144 patch_prober.go:29] interesting pod/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 00:41:52.335713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.334063 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 00:41:52.885840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.885816 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" Nov 03 00:41:52.887735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.887694 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k" event={"ID":"b3499f53-e055-4cc9-96a5-8a28e18dc3b9","Type":"ContainerDied","Data":"a6b6f708af83378426e720827b01948eb432bad410f0e9a3a6f8f5277a820948"} Nov 03 00:41:52.887838 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:52.887794 5144 scope.go:122] "RemoveContainer" containerID="eb38b2911e2feb3bafbcf813c7a628f529c2a2606b5898e339435e00ee2b648c" Nov 03 00:41:53.055211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:53.055167 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k"] Nov 03 00:41:53.058744 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:53.058717 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-jq76k"] Nov 03 00:41:53.331498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:53.331465 5144 patch_prober.go:29] interesting pod/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 00:41:53.331788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:53.331760 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 00:41:54.003899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:54.003866 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" path="/var/lib/kubelet/pods/b3499f53-e055-4cc9-96a5-8a28e18dc3b9/volumes" Nov 03 00:41:54.333168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:54.333069 5144 patch_prober.go:29] interesting pod/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" start-of-body= Nov 03 00:41:54.333436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:54.333408 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.198:15021/healthz/ready\": dial tcp 10.132.0.198:15021: connect: connection refused" Nov 03 00:41:55.332836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:55.332803 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:41:55.333239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:55.333170 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:41:55.333891 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:55.333869 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:41:58.405585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.405543 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd"] Nov 03 00:41:58.406401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406331 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" Nov 03 00:41:58.406401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406356 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406425 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406433 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406464 5144 cpu_manager.go:414] "containerMap: removing container" podUID="06f627b6-06f4-4745-97f1-32ffeda2a032" containerName="payload-pre-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406473 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f627b6-06f4-4745-97f1-32ffeda2a032" containerName="payload-pre-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406487 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" containerName="payload-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406494 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" containerName="payload-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406508 5144 cpu_manager.go:414] "containerMap: removing container" podUID="97f0b73c-20cc-45bd-8bdc-cce59b1619f1" containerName="revoke-keys" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406515 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97f0b73c-20cc-45bd-8bdc-cce59b1619f1" containerName="revoke-keys" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406526 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406533 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406552 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8edc800a-6aee-4bdb-940d-83ee6312bbf1" containerName="payload-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406560 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8edc800a-6aee-4bdb-940d-83ee6312bbf1" containerName="payload-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406570 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" containerName="payload-pre-processing" Nov 03 00:41:58.406576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406578 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" containerName="payload-pre-processing" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406760 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406782 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d75a7a8-7808-4e76-94b1-be98213ea910" containerName="istio-proxy" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406795 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f627b6-06f4-4745-97f1-32ffeda2a032" containerName="payload-pre-processing" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406804 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c29fecd-d6b1-4d1d-b561-9bcac32b83a7" containerName="payload-processing" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406814 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="97f0b73c-20cc-45bd-8bdc-cce59b1619f1" containerName="revoke-keys" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406823 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3499f53-e055-4cc9-96a5-8a28e18dc3b9" containerName="istio-proxy" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406833 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8edc800a-6aee-4bdb-940d-83ee6312bbf1" containerName="payload-processing" Nov 03 00:41:58.407285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.406841 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff98dc40-87e8-42c9-8b07-8d7f478520a8" containerName="payload-pre-processing" Nov 03 00:41:58.410464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.410441 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.470117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.470082 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd"] Nov 03 00:41:58.508814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.508772 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729j2\" (UniqueName: \"kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.509052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.508965 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.509052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.509044 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.510793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.510769 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:41:58.510917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.510865 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:41:58.610506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.610476 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.610694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.610586 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729j2\" (UniqueName: \"kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.610765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.610695 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.611428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.611356 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.613147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.613122 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.668042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.667964 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729j2\" (UniqueName: \"kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2\") pod \"maas-api-revoke-sub-33286da1-v5vmd\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:58.724104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:58.724075 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:41:59.102394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:59.102369 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd"] Nov 03 00:41:59.103273 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:41:59.103244 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cd1bf3e_b3b4_48a3_815d_8330dae28947.slice/crio-54fdcc31b6564357a9aa9e8c5abbe96cf0d7352ef09c94a85b3b33c7204397b7 WatchSource:0}: Error finding container 54fdcc31b6564357a9aa9e8c5abbe96cf0d7352ef09c94a85b3b33c7204397b7: Status 404 returned error can't find the container with id 54fdcc31b6564357a9aa9e8c5abbe96cf0d7352ef09c94a85b3b33c7204397b7 Nov 03 00:41:59.944945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:59.944900 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" event={"ID":"9cd1bf3e-b3b4-48a3-815d-8330dae28947","Type":"ContainerStarted","Data":"3302488a0a62d1cb10a4b28a1d390e8849f498f3dfb6a81634476202bafc6894"} Nov 03 00:41:59.945362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:41:59.944963 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" event={"ID":"9cd1bf3e-b3b4-48a3-815d-8330dae28947","Type":"ContainerStarted","Data":"54fdcc31b6564357a9aa9e8c5abbe96cf0d7352ef09c94a85b3b33c7204397b7"} Nov 03 00:42:00.058759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:00.058709 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" podStartSLOduration=2.058695644 podStartE2EDuration="2.058695644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:00.05769881 +0000 UTC m=+2210.203393284" watchObservedRunningTime="2026-11-03 00:42:00.058695644 +0000 UTC m=+2210.204390100" Nov 03 00:42:00.953499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:00.953459 5144 generic.go:356] "Generic (PLEG): container finished" podID="9cd1bf3e-b3b4-48a3-815d-8330dae28947" containerID="3302488a0a62d1cb10a4b28a1d390e8849f498f3dfb6a81634476202bafc6894" exitCode=0 Nov 03 00:42:00.953967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:00.953543 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" event={"ID":"9cd1bf3e-b3b4-48a3-815d-8330dae28947","Type":"ContainerDied","Data":"3302488a0a62d1cb10a4b28a1d390e8849f498f3dfb6a81634476202bafc6894"} Nov 03 00:42:02.124956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.124927 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:42:02.256272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.256240 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2\" (UniqueName: \"kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2\") pod \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " Nov 03 00:42:02.256498 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.256476 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca\") pod \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " Nov 03 00:42:02.256599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.256563 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca\") pod \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\" (UID: \"9cd1bf3e-b3b4-48a3-815d-8330dae28947\") " Nov 03 00:42:02.257006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.256968 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca" pod "9cd1bf3e-b3b4-48a3-815d-8330dae28947" (UID: "9cd1bf3e-b3b4-48a3-815d-8330dae28947"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:02.257168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.257152 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:02.259135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.259114 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2" pod "9cd1bf3e-b3b4-48a3-815d-8330dae28947" (UID: "9cd1bf3e-b3b4-48a3-815d-8330dae28947"). InnerVolumeSpecName "kube-api-access-729j2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:02.259327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.259303 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca" pod "9cd1bf3e-b3b4-48a3-815d-8330dae28947" (UID: "9cd1bf3e-b3b4-48a3-815d-8330dae28947"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:02.358237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.358208 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729j2\" (UniqueName: \"kubernetes.io/projected/9cd1bf3e-b3b4-48a3-815d-8330dae28947-kube-api-access-729j2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:02.358237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.358237 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9cd1bf3e-b3b4-48a3-815d-8330dae28947-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:02.967943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.967908 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" Nov 03 00:42:02.968128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.967952 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd" event={"ID":"9cd1bf3e-b3b4-48a3-815d-8330dae28947","Type":"ContainerDied","Data":"54fdcc31b6564357a9aa9e8c5abbe96cf0d7352ef09c94a85b3b33c7204397b7"} Nov 03 00:42:02.968128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:02.968005 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54fdcc31b6564357a9aa9e8c5abbe96cf0d7352ef09c94a85b3b33c7204397b7" Nov 03 00:42:11.528337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.528302 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2"] Nov 03 00:42:11.529199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.529179 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9cd1bf3e-b3b4-48a3-815d-8330dae28947" containerName="revoke-keys" Nov 03 00:42:11.529258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.529203 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cd1bf3e-b3b4-48a3-815d-8330dae28947" containerName="revoke-keys" Nov 03 00:42:11.529418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.529406 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cd1bf3e-b3b4-48a3-815d-8330dae28947" containerName="revoke-keys" Nov 03 00:42:11.533600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.533582 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.572146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.572112 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlfvm\" (UniqueName: \"kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.572310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.572197 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.572382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.572313 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.586430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.586403 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:11.586663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.586619 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:42:11.587832 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.587807 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2"] Nov 03 00:42:11.673592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.673555 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlfvm\" (UniqueName: \"kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.673787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.673611 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.673787 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.673691 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.674303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.674275 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.676195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.676169 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.731089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.731062 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlfvm\" (UniqueName: \"kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm\") pod \"maas-api-revoke-sub-edd15c04-cjrs2\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:11.851036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:11.850961 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:12.161304 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:12.161222 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:42:12.231227 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:42:12.231193 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9864fb78_6fa7_4c02_b38c_695fc3c65bbc.slice/crio-b8bf4a36ef48165a3a7ffda99af006c4a4d88b19e5a795f32e15bb68896d20ff WatchSource:0}: Error finding container b8bf4a36ef48165a3a7ffda99af006c4a4d88b19e5a795f32e15bb68896d20ff: Status 404 returned error can't find the container with id b8bf4a36ef48165a3a7ffda99af006c4a4d88b19e5a795f32e15bb68896d20ff Nov 03 00:42:12.231962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:12.231934 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2"] Nov 03 00:42:13.043578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.043522 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" event={"ID":"9864fb78-6fa7-4c02-b38c-695fc3c65bbc","Type":"ContainerStarted","Data":"50666c35c51940694bffb2de11ab89fe1ca6f82d00544db7f014a7b705e88d5e"} Nov 03 00:42:13.043578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.043578 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" event={"ID":"9864fb78-6fa7-4c02-b38c-695fc3c65bbc","Type":"ContainerStarted","Data":"b8bf4a36ef48165a3a7ffda99af006c4a4d88b19e5a795f32e15bb68896d20ff"} Nov 03 00:42:13.160107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.160042 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" podStartSLOduration=2.160021434 podStartE2EDuration="2.160021434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:13.156194711 +0000 UTC m=+2223.301889191" watchObservedRunningTime="2026-11-03 00:42:13.160021434 +0000 UTC m=+2223.305715890" Nov 03 00:42:13.627936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.627895 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt"] Nov 03 00:42:13.628203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.628150 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" containerID="cri-o://17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273" gracePeriod=30 Nov 03 00:42:13.675307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.675273 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh"] Nov 03 00:42:13.675483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:13.675463 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" containerID="cri-o://07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4" gracePeriod=30 Nov 03 00:42:14.051869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:14.051831 5144 generic.go:356] "Generic (PLEG): container finished" podID="9864fb78-6fa7-4c02-b38c-695fc3c65bbc" containerID="50666c35c51940694bffb2de11ab89fe1ca6f82d00544db7f014a7b705e88d5e" exitCode=0 Nov 03 00:42:14.052327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:14.051923 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" event={"ID":"9864fb78-6fa7-4c02-b38c-695fc3c65bbc","Type":"ContainerDied","Data":"50666c35c51940694bffb2de11ab89fe1ca6f82d00544db7f014a7b705e88d5e"} Nov 03 00:42:15.227253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.227220 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:15.331594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.331498 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca\") pod \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " Nov 03 00:42:15.331594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.331566 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm\" (UniqueName: \"kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm\") pod \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " Nov 03 00:42:15.331878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.331690 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca\") pod \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\" (UID: \"9864fb78-6fa7-4c02-b38c-695fc3c65bbc\") " Nov 03 00:42:15.331980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.331945 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca" pod "9864fb78-6fa7-4c02-b38c-695fc3c65bbc" (UID: "9864fb78-6fa7-4c02-b38c-695fc3c65bbc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:15.332350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.332324 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:15.333946 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.333925 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm" pod "9864fb78-6fa7-4c02-b38c-695fc3c65bbc" (UID: "9864fb78-6fa7-4c02-b38c-695fc3c65bbc"). InnerVolumeSpecName "kube-api-access-dlfvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:15.334203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.334179 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca" pod "9864fb78-6fa7-4c02-b38c-695fc3c65bbc" (UID: "9864fb78-6fa7-4c02-b38c-695fc3c65bbc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:15.434055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.434010 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlfvm\" (UniqueName: \"kubernetes.io/projected/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-kube-api-access-dlfvm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:15.434055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:15.434054 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9864fb78-6fa7-4c02-b38c-695fc3c65bbc-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:16.067920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:16.067888 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" Nov 03 00:42:16.067920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:16.067910 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2" event={"ID":"9864fb78-6fa7-4c02-b38c-695fc3c65bbc","Type":"ContainerDied","Data":"b8bf4a36ef48165a3a7ffda99af006c4a4d88b19e5a795f32e15bb68896d20ff"} Nov 03 00:42:16.068143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:16.067958 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8bf4a36ef48165a3a7ffda99af006c4a4d88b19e5a795f32e15bb68896d20ff" Nov 03 00:42:17.003340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:17.003299 5144 patch_prober.go:29] interesting pod/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:42:17.003839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:17.003384 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:42:18.984019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:18.983995 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:42:18.996213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:18.996192 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:42:19.085853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.085826 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.085865 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.085910 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.085949 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.085978 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086008 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086091 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086121 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086166 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086212 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086239 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086272 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086298 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086349 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086392 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket\") pod \"6b630523-5039-4e27-b433-8033cb75d094\" (UID: \"6b630523-5039-4e27-b433-8033cb75d094\") " Nov 03 00:42:19.086653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086432 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086549 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.086653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086586 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token\") pod \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\" (UID: \"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb\") " Nov 03 00:42:19.087095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086212 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086233 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086464 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086617 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086879 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.086926 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:19.087476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087200 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087207 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:19.087476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087384 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087432 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.087874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087855 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087881 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087895 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087908 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087923 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087936 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.087960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087950 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.088278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087963 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.088278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087978 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.088278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.087992 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b630523-5039-4e27-b433-8033cb75d094-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.088976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.088950 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.090651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.089814 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:19.090651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.089874 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:19.090651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.090252 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:19.090651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.090347 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:19.090651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.090480 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091407 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c" pod "6b630523-5039-4e27-b433-8033cb75d094" (UID: "6b630523-5039-4e27-b433-8033cb75d094"). InnerVolumeSpecName "kube-api-access-78f4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091459 5144 generic.go:356] "Generic (PLEG): container finished" podID="6b630523-5039-4e27-b433-8033cb75d094" containerID="17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273" exitCode=0 Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091507 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn" pod "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" (UID: "e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb"). InnerVolumeSpecName "kube-api-access-9g6wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091552 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" event={"ID":"6b630523-5039-4e27-b433-8033cb75d094","Type":"ContainerDied","Data":"17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273"} Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091580 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091593 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt" event={"ID":"6b630523-5039-4e27-b433-8033cb75d094","Type":"ContainerDied","Data":"1d188a920dfafc36732e7ec5ca51f20f28f94b8eca85739b2f2deba76d1f1bd7"} Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.091613 5144 scope.go:122] "RemoveContainer" containerID="17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273" Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.093716 5144 generic.go:356] "Generic (PLEG): container finished" podID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerID="07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4" exitCode=0 Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.093780 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.093794 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" event={"ID":"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb","Type":"ContainerDied","Data":"07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4"} Nov 03 00:42:19.099666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.093850 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh" event={"ID":"e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb","Type":"ContainerDied","Data":"0e18e1b851b59c7fe966e02e6bd73e042a228f26f4dc1dd8d16af7e7a7e7309e"} Nov 03 00:42:19.107732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.106855 5144 scope.go:122] "RemoveContainer" containerID="17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273" Nov 03 00:42:19.107732 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:19.107673 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273\": container with ID starting with 17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273 not found: ID does not exist" containerID="17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273" Nov 03 00:42:19.107732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.107707 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273"} err="failed to get container status \"17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273\": rpc error: code = NotFound desc = could not find container \"17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273\": container with ID starting with 17c2406832cdf2193ced88837cab76a519f875d200aa707c16de69c251493273 not found: ID does not exist" Nov 03 00:42:19.107732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.107731 5144 scope.go:122] "RemoveContainer" containerID="07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4" Nov 03 00:42:19.117761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.117737 5144 scope.go:122] "RemoveContainer" containerID="07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4" Nov 03 00:42:19.118035 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:19.118017 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4\": container with ID starting with 07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4 not found: ID does not exist" containerID="07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4" Nov 03 00:42:19.118109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.118045 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4"} err="failed to get container status \"07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4\": rpc error: code = NotFound desc = could not find container \"07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4\": container with ID starting with 07714f421c2595731602c164296879edda0d169d944473834b2c93d1d56359f4 not found: ID does not exist" Nov 03 00:42:19.189924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189833 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.189924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189868 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b630523-5039-4e27-b433-8033cb75d094-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.189924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189880 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6wn\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-kube-api-access-9g6wn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.189924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189893 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.189924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189906 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b630523-5039-4e27-b433-8033cb75d094-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.189924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189923 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78f4c\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-kube-api-access-78f4c\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.190317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189937 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b630523-5039-4e27-b433-8033cb75d094-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.190317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.189949 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:19.265052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.265022 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt"] Nov 03 00:42:19.272797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.272762 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-hrcjt"] Nov 03 00:42:19.444220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.444129 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh"] Nov 03 00:42:19.450519 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:19.450492 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-048135-openshift-default-754f957787-2p9qh"] Nov 03 00:42:20.004975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:20.001897 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b630523-5039-4e27-b433-8033cb75d094" path="/var/lib/kubelet/pods/6b630523-5039-4e27-b433-8033cb75d094/volumes" Nov 03 00:42:20.004975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:20.002694 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" path="/var/lib/kubelet/pods/e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb/volumes" Nov 03 00:42:28.107885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.107852 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt"] Nov 03 00:42:28.108448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108435 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" Nov 03 00:42:28.108488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108449 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" Nov 03 00:42:28.108488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108461 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" Nov 03 00:42:28.108488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108466 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" Nov 03 00:42:28.108488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108487 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9864fb78-6fa7-4c02-b38c-695fc3c65bbc" containerName="revoke-keys" Nov 03 00:42:28.108624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108493 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9864fb78-6fa7-4c02-b38c-695fc3c65bbc" containerName="revoke-keys" Nov 03 00:42:28.108624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108592 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6d6d3ec-41d6-4fa5-875b-b30fe3f184eb" containerName="istio-proxy" Nov 03 00:42:28.108624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108604 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9864fb78-6fa7-4c02-b38c-695fc3c65bbc" containerName="revoke-keys" Nov 03 00:42:28.108624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.108616 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b630523-5039-4e27-b433-8033cb75d094" containerName="istio-proxy" Nov 03 00:42:28.113608 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.113592 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.170056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.170031 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt"] Nov 03 00:42:28.214696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.214651 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:28.214832 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.214734 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:42:28.306470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.306439 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2dfc\" (UniqueName: \"kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.306608 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.306508 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.306758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.306741 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.408265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.408200 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.408399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.408263 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2dfc\" (UniqueName: \"kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.408399 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.408303 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.408874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.408853 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.410808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.410791 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.470241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.470208 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2dfc\" (UniqueName: \"kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc\") pod \"maas-api-revoke-sub-68eda06d-tbfgt\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:28.726980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:28.726899 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:29.105168 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:42:29.105140 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80abe2c6_9de2_4dc7_96e0_ea92c5deee13.slice/crio-d3210debeb3305c2dcb0408bfcbc1f28d6e1b203c6d80981f736d9d6dbea8e94 WatchSource:0}: Error finding container d3210debeb3305c2dcb0408bfcbc1f28d6e1b203c6d80981f736d9d6dbea8e94: Status 404 returned error can't find the container with id d3210debeb3305c2dcb0408bfcbc1f28d6e1b203c6d80981f736d9d6dbea8e94 Nov 03 00:42:29.105316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:29.105245 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt"] Nov 03 00:42:29.178428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:29.178327 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" event={"ID":"80abe2c6-9de2-4dc7-96e0-ea92c5deee13","Type":"ContainerStarted","Data":"d3210debeb3305c2dcb0408bfcbc1f28d6e1b203c6d80981f736d9d6dbea8e94"} Nov 03 00:42:30.185901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:30.185864 5144 generic.go:356] "Generic (PLEG): container finished" podID="80abe2c6-9de2-4dc7-96e0-ea92c5deee13" containerID="77392348e73fe7e08ab6fff46a146b6c118c8f9f93ee39e60041fbdb91af0685" exitCode=0 Nov 03 00:42:30.186255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:30.185938 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" event={"ID":"80abe2c6-9de2-4dc7-96e0-ea92c5deee13","Type":"ContainerDied","Data":"77392348e73fe7e08ab6fff46a146b6c118c8f9f93ee39e60041fbdb91af0685"} Nov 03 00:42:31.345484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.345459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:31.447457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.447375 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca\") pod \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " Nov 03 00:42:31.447457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.447432 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca\") pod \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " Nov 03 00:42:31.447723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.447470 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc\" (UniqueName: \"kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc\") pod \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\" (UID: \"80abe2c6-9de2-4dc7-96e0-ea92c5deee13\") " Nov 03 00:42:31.447845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.447816 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca" pod "80abe2c6-9de2-4dc7-96e0-ea92c5deee13" (UID: "80abe2c6-9de2-4dc7-96e0-ea92c5deee13"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:31.448152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.448132 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:31.450210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.450180 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca" pod "80abe2c6-9de2-4dc7-96e0-ea92c5deee13" (UID: "80abe2c6-9de2-4dc7-96e0-ea92c5deee13"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:31.450317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.450264 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc" pod "80abe2c6-9de2-4dc7-96e0-ea92c5deee13" (UID: "80abe2c6-9de2-4dc7-96e0-ea92c5deee13"). InnerVolumeSpecName "kube-api-access-w2dfc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:31.549154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.549122 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:31.549154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:31.549153 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2dfc\" (UniqueName: \"kubernetes.io/projected/80abe2c6-9de2-4dc7-96e0-ea92c5deee13-kube-api-access-w2dfc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:32.032197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.032157 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd"] Nov 03 00:42:32.032847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.032827 5144 cpu_manager.go:414] "containerMap: removing container" podUID="80abe2c6-9de2-4dc7-96e0-ea92c5deee13" containerName="revoke-keys" Nov 03 00:42:32.032958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.032848 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80abe2c6-9de2-4dc7-96e0-ea92c5deee13" containerName="revoke-keys" Nov 03 00:42:32.033044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.033030 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="80abe2c6-9de2-4dc7-96e0-ea92c5deee13" containerName="revoke-keys" Nov 03 00:42:32.036534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.036515 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.055199 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.055171 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.055352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.055223 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.055352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.055248 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.055472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.055378 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.055472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.055426 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kfq8\" (UniqueName: \"kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.055577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.055549 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.097188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.097151 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd"] Nov 03 00:42:32.138591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.138545 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d61eae\"/\"kube-root-ca.crt\"" Nov 03 00:42:32.138591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.138545 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-d61eae\"/\"openshift-service-ca.crt\"" Nov 03 00:42:32.138865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.138565 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d61eae\"/\"default-dockercfg-8ksml\"" Nov 03 00:42:32.138865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.138621 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-d61eae\"/\"praxis-route-d61eae-kserve-self-signed-certs\"" Nov 03 00:42:32.156179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156150 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156201 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kfq8\" (UniqueName: \"kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156284 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156387 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156480 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156453 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156490 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156575 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156833 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.156972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.156889 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.158772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.158748 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.159028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.159010 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.202211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.202178 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" Nov 03 00:42:32.202408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.202206 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt" event={"ID":"80abe2c6-9de2-4dc7-96e0-ea92c5deee13","Type":"ContainerDied","Data":"d3210debeb3305c2dcb0408bfcbc1f28d6e1b203c6d80981f736d9d6dbea8e94"} Nov 03 00:42:32.202408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.202258 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3210debeb3305c2dcb0408bfcbc1f28d6e1b203c6d80981f736d9d6dbea8e94" Nov 03 00:42:32.212456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.212427 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kfq8\" (UniqueName: \"kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8\") pod \"praxis-route-d61eae-kserve-76bccc49cc-svlfd\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.349378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.349304 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:32.725816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:32.725788 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd"] Nov 03 00:42:32.726900 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:42:32.726864 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod861a520b_3348_4f6a_8fc6_ae574119b96e.slice/crio-f7e557edf833ff99607825a3a6646873aa5885a3ab330e38544a570bf0620aca WatchSource:0}: Error finding container f7e557edf833ff99607825a3a6646873aa5885a3ab330e38544a570bf0620aca: Status 404 returned error can't find the container with id f7e557edf833ff99607825a3a6646873aa5885a3ab330e38544a570bf0620aca Nov 03 00:42:33.211181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:33.211148 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" event={"ID":"861a520b-3348-4f6a-8fc6-ae574119b96e","Type":"ContainerStarted","Data":"4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c"} Nov 03 00:42:33.211351 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:33.211194 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" event={"ID":"861a520b-3348-4f6a-8fc6-ae574119b96e","Type":"ContainerStarted","Data":"f7e557edf833ff99607825a3a6646873aa5885a3ab330e38544a570bf0620aca"} Nov 03 00:42:33.349692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:33.349649 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:42.155285 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:42.155245 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:42:42.349473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:42.349435 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:42.362877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:42.362849 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:42.478813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:42.478703 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" podStartSLOduration=11.47868778 podStartE2EDuration="11.47868778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:33.324570941 +0000 UTC m=+2243.470265399" watchObservedRunningTime="2026-11-03 00:42:42.47868778 +0000 UTC m=+2252.624382236" Nov 03 00:42:43.305775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.305734 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:42:43.635481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.635389 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k"] Nov 03 00:42:43.635700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.635669 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" containerID="cri-o://23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72" gracePeriod=30 Nov 03 00:42:43.643033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.642166 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5"] Nov 03 00:42:43.643033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.642424 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" containerID="cri-o://4ab4867f9633352ad9531c6805bb29d4443ec515cf54b4627a2e60528d9c3492" gracePeriod=30 Nov 03 00:42:43.648568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.648547 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq"] Nov 03 00:42:43.651654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.651612 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" containerID="cri-o://d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0" gracePeriod=30 Nov 03 00:42:43.682314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.682283 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k"] Nov 03 00:42:43.682492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:43.682470 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" containerID="cri-o://44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80" gracePeriod=30 Nov 03 00:42:44.293706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.293670 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj"] Nov 03 00:42:44.298427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.298405 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.354622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.354592 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:44.354980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.354679 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:42:44.355751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.355727 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj"] Nov 03 00:42:44.439662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.439616 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.439803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.439767 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgb2l\" (UniqueName: \"kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.439884 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.439871 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.540545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.540516 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.540757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.540662 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgb2l\" (UniqueName: \"kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.540757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.540716 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.541246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.541208 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.543444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.543417 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.598898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.598838 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgb2l\" (UniqueName: \"kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l\") pod \"maas-api-revoke-sub-daab9573-mwhqj\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.612492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.612469 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:44.989062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:44.989036 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj"] Nov 03 00:42:44.989854 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:42:44.989828 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f0f439_2e96_4093_acdd_1d57446f4b63.slice/crio-ff92e0ce0a37f579a1fc1f50ce7104f2350efe8b1e8cdf95d0d9f245f66818fe WatchSource:0}: Error finding container ff92e0ce0a37f579a1fc1f50ce7104f2350efe8b1e8cdf95d0d9f245f66818fe: Status 404 returned error can't find the container with id ff92e0ce0a37f579a1fc1f50ce7104f2350efe8b1e8cdf95d0d9f245f66818fe Nov 03 00:42:45.308438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:45.308413 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" event={"ID":"82f0f439-2e96-4093-acdd-1d57446f4b63","Type":"ContainerStarted","Data":"ff92e0ce0a37f579a1fc1f50ce7104f2350efe8b1e8cdf95d0d9f245f66818fe"} Nov 03 00:42:46.317073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:46.317042 5144 generic.go:356] "Generic (PLEG): container finished" podID="82f0f439-2e96-4093-acdd-1d57446f4b63" containerID="9b01f62b09f56c99d8e19263fa97fea9e2d0684f63b422246fa17195b326de72" exitCode=0 Nov 03 00:42:46.317460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:46.317133 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" event={"ID":"82f0f439-2e96-4093-acdd-1d57446f4b63","Type":"ContainerDied","Data":"9b01f62b09f56c99d8e19263fa97fea9e2d0684f63b422246fa17195b326de72"} Nov 03 00:42:46.489175 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:46.489142 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:42:46.489345 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:46.489204 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:42:47.197295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.197263 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf"] Nov 03 00:42:47.201386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.201368 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266262 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266221 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266467 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266528 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266623 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266714 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266756 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266832 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266936 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.266988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.266970 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdcmt\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.306032 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.306001 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-531ee4-openshift-default-dockercfg-pcjd8\"" Nov 03 00:42:47.309388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.309364 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf"] Nov 03 00:42:47.368957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.368919 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.368966 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdcmt\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369051 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369069 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369099 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369114 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369141 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369155 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369175 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.369724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.369550 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.370196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.370170 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.370317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.370299 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.370509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.370494 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.370575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.370489 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.372575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.372543 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.373248 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.373228 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.429676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.427298 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdcmt\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.429676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.428254 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.490615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.490590 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:47.516143 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.516120 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:47.570914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.570883 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l\" (UniqueName: \"kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l\") pod \"82f0f439-2e96-4093-acdd-1d57446f4b63\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " Nov 03 00:42:47.571093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.570941 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca\") pod \"82f0f439-2e96-4093-acdd-1d57446f4b63\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " Nov 03 00:42:47.571093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.571075 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca\") pod \"82f0f439-2e96-4093-acdd-1d57446f4b63\" (UID: \"82f0f439-2e96-4093-acdd-1d57446f4b63\") " Nov 03 00:42:47.571360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.571327 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca" pod "82f0f439-2e96-4093-acdd-1d57446f4b63" (UID: "82f0f439-2e96-4093-acdd-1d57446f4b63"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:47.572532 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.572286 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:47.573907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.573870 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca" pod "82f0f439-2e96-4093-acdd-1d57446f4b63" (UID: "82f0f439-2e96-4093-acdd-1d57446f4b63"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:47.574144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.574120 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l" pod "82f0f439-2e96-4093-acdd-1d57446f4b63" (UID: "82f0f439-2e96-4093-acdd-1d57446f4b63"). InnerVolumeSpecName "kube-api-access-zgb2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:47.673344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.673302 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgb2l\" (UniqueName: \"kubernetes.io/projected/82f0f439-2e96-4093-acdd-1d57446f4b63-kube-api-access-zgb2l\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:47.673344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.673339 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82f0f439-2e96-4093-acdd-1d57446f4b63-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:47.945793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.945767 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf"] Nov 03 00:42:47.947359 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:42:47.947333 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcb458a4_e6dd_41c8_84e9_da59100fcb49.slice/crio-a6ea1ec56b5592966c8e48c4fa5737231dd6b7fd655fa29d2b4cd9e7792ecac1 WatchSource:0}: Error finding container a6ea1ec56b5592966c8e48c4fa5737231dd6b7fd655fa29d2b4cd9e7792ecac1: Status 404 returned error can't find the container with id a6ea1ec56b5592966c8e48c4fa5737231dd6b7fd655fa29d2b4cd9e7792ecac1 Nov 03 00:42:47.950446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.950411 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:42:47.950524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.950491 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:42:47.950559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:47.950532 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:42:48.348045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.347772 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" event={"ID":"bcb458a4-e6dd-41c8-84e9-da59100fcb49","Type":"ContainerStarted","Data":"35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02"} Nov 03 00:42:48.348045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.347852 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" event={"ID":"bcb458a4-e6dd-41c8-84e9-da59100fcb49","Type":"ContainerStarted","Data":"a6ea1ec56b5592966c8e48c4fa5737231dd6b7fd655fa29d2b4cd9e7792ecac1"} Nov 03 00:42:48.349732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.349704 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" Nov 03 00:42:48.350583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.350551 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj" event={"ID":"82f0f439-2e96-4093-acdd-1d57446f4b63","Type":"ContainerDied","Data":"ff92e0ce0a37f579a1fc1f50ce7104f2350efe8b1e8cdf95d0d9f245f66818fe"} Nov 03 00:42:48.350772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.350604 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ff92e0ce0a37f579a1fc1f50ce7104f2350efe8b1e8cdf95d0d9f245f66818fe" Nov 03 00:42:48.463454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.463403 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podStartSLOduration=1.463389366 podStartE2EDuration="1.463389366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:48.462926134 +0000 UTC m=+2258.608620630" watchObservedRunningTime="2026-11-03 00:42:48.463389366 +0000 UTC m=+2258.609083822" Nov 03 00:42:48.516647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:48.516600 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:49.059733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.059619 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:42:49.092244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092212 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092299 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092336 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092423 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092407 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092458 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092502 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092537 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092572 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.092795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.092618 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert\") pod \"fd8e32b9-f553-406c-a3bb-38cb42456664\" (UID: \"fd8e32b9-f553-406c-a3bb-38cb42456664\") " Nov 03 00:42:49.093243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.093196 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.093403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.093373 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.093835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.093781 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.095842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.095504 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.095842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.095752 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.096099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.096066 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:49.100099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.100072 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:49.100868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.100850 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.102715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.102691 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "kube-api-access-vmp4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.103620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.103597 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy" pod "fd8e32b9-f553-406c-a3bb-38cb42456664" (UID: "fd8e32b9-f553-406c-a3bb-38cb42456664"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.175793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.175757 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:42:49.180516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.180308 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:42:49.194500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.194455 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.194729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.194691 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.195137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195117 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.195327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195294 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195035 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195423 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195497 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195565 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195599 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195642 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.195757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195714 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.196191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195836 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.196191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.195855 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.196301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196287 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196449 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196560 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196649 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196695 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196741 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196777 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196816 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs\") pod \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\" (UID: \"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88\") " Nov 03 00:42:49.196945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.196854 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.197672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.197557 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert\") pod \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\" (UID: \"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee\") " Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198303 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmp4f\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-kube-api-access-vmp4f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198333 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198347 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198368 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198383 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd8e32b9-f553-406c-a3bb-38cb42456664-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198397 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198412 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198425 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198438 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198452 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198466 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198479 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198492 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd8e32b9-f553-406c-a3bb-38cb42456664-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.198693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198648 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:49.199484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.198984 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:49.200057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.199782 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.200057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.199783 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:49.200057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.200099 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.200057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.200308 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.200057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.200982 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.201790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.201755 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:49.202060 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.202035 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "kube-api-access-8frpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.202287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.202238 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.203335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.203307 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.204531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.204498 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg" pod "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" (UID: "dbde53d5-8b8d-49a2-b9eb-b8a852a60e88"). InnerVolumeSpecName "kube-api-access-4gzbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.204697 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.204651 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token" pod "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" (UID: "d8404df7-e0e6-4fb9-b3c1-2992a602e0ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.274132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.268431 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-995g5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 00:42:49.274132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.268496 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 00:42:49.300218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300170 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.300437 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300418 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.300558 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300544 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.300683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300667 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.300773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300761 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.300871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300859 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.300966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.300953 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8frpv\" (UniqueName: \"kubernetes.io/projected/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-kube-api-access-8frpv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.301058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.301047 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.301149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.301138 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.301230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.301219 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.301316 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.301305 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.301403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.301392 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.301484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.301472 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gzbg\" (UniqueName: \"kubernetes.io/projected/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88-kube-api-access-4gzbg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.369594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.369560 5144 generic.go:356] "Generic (PLEG): container finished" podID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerID="23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72" exitCode=0 Nov 03 00:42:49.369789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.369760 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" Nov 03 00:42:49.371656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.371352 5144 generic.go:356] "Generic (PLEG): container finished" podID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerID="44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80" exitCode=0 Nov 03 00:42:49.371656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.371502 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" Nov 03 00:42:49.375978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.375826 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" event={"ID":"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee","Type":"ContainerDied","Data":"23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72"} Nov 03 00:42:49.376104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.375980 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k" event={"ID":"d8404df7-e0e6-4fb9-b3c1-2992a602e0ee","Type":"ContainerDied","Data":"0fc3fc5710f6302745041fe3b2d00556f0e5e38844bf84ffcd2247baf6519970"} Nov 03 00:42:49.376104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.376013 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" event={"ID":"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88","Type":"ContainerDied","Data":"44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80"} Nov 03 00:42:49.376104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.376064 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k" event={"ID":"dbde53d5-8b8d-49a2-b9eb-b8a852a60e88","Type":"ContainerDied","Data":"bc61695167a6a56a9340ded55180eeebfe49d92c40b17bad96c5f618a15ce2db"} Nov 03 00:42:49.376104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.376097 5144 scope.go:122] "RemoveContainer" containerID="23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72" Nov 03 00:42:49.379517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.379499 5144 generic.go:356] "Generic (PLEG): container finished" podID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerID="d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0" exitCode=0 Nov 03 00:42:49.379680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.379577 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" Nov 03 00:42:49.379880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.379587 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" event={"ID":"fd8e32b9-f553-406c-a3bb-38cb42456664","Type":"ContainerDied","Data":"d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0"} Nov 03 00:42:49.380004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.379989 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq" event={"ID":"fd8e32b9-f553-406c-a3bb-38cb42456664","Type":"ContainerDied","Data":"78b24f25566036ecdac05be1b3285bd3454b29c44757e93f4fefddfca6085a75"} Nov 03 00:42:49.382202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.382182 5144 generic.go:356] "Generic (PLEG): container finished" podID="eaa60f82-9e28-423b-8788-0b28834b7052" containerID="4ab4867f9633352ad9531c6805bb29d4443ec515cf54b4627a2e60528d9c3492" exitCode=0 Nov 03 00:42:49.383094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.383070 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" event={"ID":"eaa60f82-9e28-423b-8788-0b28834b7052","Type":"ContainerDied","Data":"4ab4867f9633352ad9531c6805bb29d4443ec515cf54b4627a2e60528d9c3492"} Nov 03 00:42:49.463466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.463375 5144 scope.go:122] "RemoveContainer" containerID="23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72" Nov 03 00:42:49.463869 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:49.463779 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72\": container with ID starting with 23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72 not found: ID does not exist" containerID="23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72" Nov 03 00:42:49.463869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.463817 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72"} err="failed to get container status \"23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72\": rpc error: code = NotFound desc = could not find container \"23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72\": container with ID starting with 23bc284cffdfd64bb35de5530313dc0340a395cb29768e3468e56a7ba1868f72 not found: ID does not exist" Nov 03 00:42:49.463869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.463843 5144 scope.go:122] "RemoveContainer" containerID="44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80" Nov 03 00:42:49.468255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.468236 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:42:49.477082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.477072 5144 scope.go:122] "RemoveContainer" containerID="44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80" Nov 03 00:42:49.477358 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:49.477338 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80\": container with ID starting with 44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80 not found: ID does not exist" containerID="44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80" Nov 03 00:42:49.477440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.477363 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80"} err="failed to get container status \"44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80\": rpc error: code = NotFound desc = could not find container \"44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80\": container with ID starting with 44cf53ec258ecaf14c291ee0f8ed6d3713b7cbab965b2aac3f2c32d2361cbe80 not found: ID does not exist" Nov 03 00:42:49.477440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.477391 5144 scope.go:122] "RemoveContainer" containerID="d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0" Nov 03 00:42:49.488916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.488676 5144 scope.go:122] "RemoveContainer" containerID="d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0" Nov 03 00:42:49.489202 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:42:49.489108 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0\": container with ID starting with d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0 not found: ID does not exist" containerID="d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0" Nov 03 00:42:49.489202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.489143 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0"} err="failed to get container status \"d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0\": rpc error: code = NotFound desc = could not find container \"d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0\": container with ID starting with d34991101d54723c6437c7f6199e31e02895a6d67f621633e675136c15b239a0 not found: ID does not exist" Nov 03 00:42:49.502999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.502973 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503061 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503094 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503194 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503240 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503303 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503360 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503415 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.503515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503445 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token\") pod \"eaa60f82-9e28-423b-8788-0b28834b7052\" (UID: \"eaa60f82-9e28-423b-8788-0b28834b7052\") " Nov 03 00:42:49.504104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.503916 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.504602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.504337 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.504602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.504503 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.504602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.504569 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:49.504927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.504691 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.506732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.506687 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "kube-api-access-qwh57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.506811 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.506725 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:42:49.507903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.507881 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:42:49.508517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.508493 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token" pod "eaa60f82-9e28-423b-8788-0b28834b7052" (UID: "eaa60f82-9e28-423b-8788-0b28834b7052"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:49.552288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.552238 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:42:49.552487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.552297 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:42:49.604903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604873 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.604903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604899 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwh57\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-kube-api-access-qwh57\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.604903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604909 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaa60f82-9e28-423b-8788-0b28834b7052-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.605132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604918 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.605132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604926 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaa60f82-9e28-423b-8788-0b28834b7052-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.605132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604934 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.605132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604943 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaa60f82-9e28-423b-8788-0b28834b7052-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.605132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604951 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.605132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.604961 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaa60f82-9e28-423b-8788-0b28834b7052-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:49.759152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.759116 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k"] Nov 03 00:42:49.770323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.770289 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-rln6k"] Nov 03 00:42:49.885397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.885292 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k"] Nov 03 00:42:49.888730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.888703 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-bln8k"] Nov 03 00:42:49.998149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.998118 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" path="/var/lib/kubelet/pods/d8404df7-e0e6-4fb9-b3c1-2992a602e0ee/volumes" Nov 03 00:42:49.998725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:49.998693 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" path="/var/lib/kubelet/pods/dbde53d5-8b8d-49a2-b9eb-b8a852a60e88/volumes" Nov 03 00:42:50.003906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.003882 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq"] Nov 03 00:42:50.006923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.006903 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-xsmvq"] Nov 03 00:42:50.394668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.394624 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" Nov 03 00:42:50.394668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.394658 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5" event={"ID":"eaa60f82-9e28-423b-8788-0b28834b7052","Type":"ContainerDied","Data":"0a4238f47706d2bd46a8883c5e6817cfa285bab90067e22a0b5e2ed16dd4adea"} Nov 03 00:42:50.394929 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.394723 5144 scope.go:122] "RemoveContainer" containerID="4ab4867f9633352ad9531c6805bb29d4443ec515cf54b4627a2e60528d9c3492" Nov 03 00:42:50.520660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.517986 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:42:50.520660 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.518040 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:42:50.561421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.561378 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5"] Nov 03 00:42:50.564501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:50.564471 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-995g5"] Nov 03 00:42:51.522658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:51.519989 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:42:51.522658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:51.520052 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:42:51.996125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:51.996053 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" path="/var/lib/kubelet/pods/eaa60f82-9e28-423b-8788-0b28834b7052/volumes" Nov 03 00:42:51.996475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:51.996462 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" path="/var/lib/kubelet/pods/fd8e32b9-f553-406c-a3bb-38cb42456664/volumes" Nov 03 00:42:52.521658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:52.518971 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:42:52.521658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:52.519032 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:42:53.532657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:53.519938 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:42:53.532657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:53.520028 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:42:54.394334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394302 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q"] Nov 03 00:42:54.394900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394883 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" Nov 03 00:42:54.394900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394898 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394912 5144 cpu_manager.go:414] "containerMap: removing container" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394916 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394933 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394937 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394954 5144 cpu_manager.go:414] "containerMap: removing container" podUID="82f0f439-2e96-4093-acdd-1d57446f4b63" containerName="revoke-keys" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394958 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f0f439-2e96-4093-acdd-1d57446f4b63" containerName="revoke-keys" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394978 5144 cpu_manager.go:414] "containerMap: removing container" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" Nov 03 00:42:54.395101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.394982 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" Nov 03 00:42:54.395382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.395105 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd8e32b9-f553-406c-a3bb-38cb42456664" containerName="istio-proxy" Nov 03 00:42:54.395382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.395123 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8404df7-e0e6-4fb9-b3c1-2992a602e0ee" containerName="istio-proxy" Nov 03 00:42:54.395382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.395137 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa60f82-9e28-423b-8788-0b28834b7052" containerName="istio-proxy" Nov 03 00:42:54.395382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.395155 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbde53d5-8b8d-49a2-b9eb-b8a852a60e88" containerName="istio-proxy" Nov 03 00:42:54.395382 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.395168 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f0f439-2e96-4093-acdd-1d57446f4b63" containerName="revoke-keys" Nov 03 00:42:54.400909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.400889 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.477850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.477816 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlg8t\" (UniqueName: \"kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.478089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.477922 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.478089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.477954 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.509285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.509258 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:42:54.509444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.509337 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:42:54.512908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.512888 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q"] Nov 03 00:42:54.517008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.516989 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" start-of-body= Nov 03 00:42:54.517081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.517037 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.205:15021/healthz/ready\": dial tcp 10.132.0.205:15021: connect: connection refused" Nov 03 00:42:54.579544 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.579516 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlg8t\" (UniqueName: \"kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.579866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.579657 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.579866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.579697 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.580336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.580310 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.582207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.582186 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.636342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.636323 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlg8t\" (UniqueName: \"kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t\") pod \"maas-api-revoke-sub-dc8e1f58-jwl2q\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:54.714611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:54.714552 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:55.091241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:55.091132 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q"] Nov 03 00:42:55.454056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:55.453971 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" event={"ID":"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1","Type":"ContainerStarted","Data":"473f5ee50da20d60738265513e23461401b4412a6df1cd409ea846d6d1931630"} Nov 03 00:42:55.521046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:55.521017 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:55.521486 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:55.521460 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:55.522289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:55.522270 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:42:56.461914 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:56.461825 5144 generic.go:356] "Generic (PLEG): container finished" podID="65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" containerID="85d37b9e44fff045aa4e956fb1e311a967bf632cd44ee9b8c78ad0a9abec64a3" exitCode=0 Nov 03 00:42:56.462387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:56.461908 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" event={"ID":"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1","Type":"ContainerDied","Data":"85d37b9e44fff045aa4e956fb1e311a967bf632cd44ee9b8c78ad0a9abec64a3"} Nov 03 00:42:57.623985 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.623956 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:57.719454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.719395 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t\" (UniqueName: \"kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t\") pod \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " Nov 03 00:42:57.719580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.719470 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca\") pod \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " Nov 03 00:42:57.719580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.719515 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca\") pod \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\" (UID: \"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1\") " Nov 03 00:42:57.719905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.719882 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca" pod "65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" (UID: "65a81826-7db1-4b9a-9b2e-3132d4bf9fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:42:57.721478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.721453 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t" pod "65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" (UID: "65a81826-7db1-4b9a-9b2e-3132d4bf9fe1"). InnerVolumeSpecName "kube-api-access-mlg8t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:42:57.721593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.721535 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca" pod "65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" (UID: "65a81826-7db1-4b9a-9b2e-3132d4bf9fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:42:57.821394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.821360 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:57.821394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.821393 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:57.821591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:57.821409 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlg8t\" (UniqueName: \"kubernetes.io/projected/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1-kube-api-access-mlg8t\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:42:58.476835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:58.476755 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" Nov 03 00:42:58.476984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:58.476754 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q" event={"ID":"65a81826-7db1-4b9a-9b2e-3132d4bf9fe1","Type":"ContainerDied","Data":"473f5ee50da20d60738265513e23461401b4412a6df1cd409ea846d6d1931630"} Nov 03 00:42:58.476984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:42:58.476871 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="473f5ee50da20d60738265513e23461401b4412a6df1cd409ea846d6d1931630" Nov 03 00:43:00.626839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.626809 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b"] Nov 03 00:43:00.627370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.627357 5144 cpu_manager.go:414] "containerMap: removing container" podUID="65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" containerName="revoke-keys" Nov 03 00:43:00.627425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.627371 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" containerName="revoke-keys" Nov 03 00:43:00.627472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.627463 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" containerName="revoke-keys" Nov 03 00:43:00.631051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.631032 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.687436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.687403 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b"] Nov 03 00:43:00.732479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.732450 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:00.732698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.732494 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:43:00.749788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.749759 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzgg8\" (UniqueName: \"kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.749944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.749926 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.750007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.749989 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.851301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.851273 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzgg8\" (UniqueName: \"kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.851491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.851375 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.851491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.851424 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.852088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.852065 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.853758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.853739 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.908142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.908078 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzgg8\" (UniqueName: \"kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8\") pod \"maas-api-revoke-sub-095dd76a-rhz8b\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:00.948331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:00.948307 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:01.323258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:01.323230 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b"] Nov 03 00:43:01.323977 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:01.323950 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec55f256_9d9d_4232_a41e_f61e0aea18df.slice/crio-40ec180c57c0e1777578f60e4d701812abc228881f23b4b45a4dcd7f619446e2 WatchSource:0}: Error finding container 40ec180c57c0e1777578f60e4d701812abc228881f23b4b45a4dcd7f619446e2: Status 404 returned error can't find the container with id 40ec180c57c0e1777578f60e4d701812abc228881f23b4b45a4dcd7f619446e2 Nov 03 00:43:01.496881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:01.496846 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" event={"ID":"ec55f256-9d9d-4232-a41e-f61e0aea18df","Type":"ContainerStarted","Data":"40ec180c57c0e1777578f60e4d701812abc228881f23b4b45a4dcd7f619446e2"} Nov 03 00:43:02.505277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:02.505247 5144 generic.go:356] "Generic (PLEG): container finished" podID="ec55f256-9d9d-4232-a41e-f61e0aea18df" containerID="7980b897ce2a35a883658421102e85b6a993251670ec5aac25263e6778db338c" exitCode=0 Nov 03 00:43:02.505723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:02.505341 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" event={"ID":"ec55f256-9d9d-4232-a41e-f61e0aea18df","Type":"ContainerDied","Data":"7980b897ce2a35a883658421102e85b6a993251670ec5aac25263e6778db338c"} Nov 03 00:43:03.708470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.708440 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:03.784089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.783992 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca\") pod \"ec55f256-9d9d-4232-a41e-f61e0aea18df\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " Nov 03 00:43:03.784270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.784176 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca\") pod \"ec55f256-9d9d-4232-a41e-f61e0aea18df\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " Nov 03 00:43:03.784270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.784240 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8\" (UniqueName: \"kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8\") pod \"ec55f256-9d9d-4232-a41e-f61e0aea18df\" (UID: \"ec55f256-9d9d-4232-a41e-f61e0aea18df\") " Nov 03 00:43:03.784509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.784485 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca" pod "ec55f256-9d9d-4232-a41e-f61e0aea18df" (UID: "ec55f256-9d9d-4232-a41e-f61e0aea18df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:03.784760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.784740 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:03.786470 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.786443 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca" pod "ec55f256-9d9d-4232-a41e-f61e0aea18df" (UID: "ec55f256-9d9d-4232-a41e-f61e0aea18df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:03.786609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.786513 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8" pod "ec55f256-9d9d-4232-a41e-f61e0aea18df" (UID: "ec55f256-9d9d-4232-a41e-f61e0aea18df"). InnerVolumeSpecName "kube-api-access-zzgg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:03.885572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.885539 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ec55f256-9d9d-4232-a41e-f61e0aea18df-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:03.885572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:03.885570 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzgg8\" (UniqueName: \"kubernetes.io/projected/ec55f256-9d9d-4232-a41e-f61e0aea18df-kube-api-access-zzgg8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:04.523947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:04.523914 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" Nov 03 00:43:04.523947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:04.523925 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b" event={"ID":"ec55f256-9d9d-4232-a41e-f61e0aea18df","Type":"ContainerDied","Data":"40ec180c57c0e1777578f60e4d701812abc228881f23b4b45a4dcd7f619446e2"} Nov 03 00:43:04.524163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:04.524011 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ec180c57c0e1777578f60e4d701812abc228881f23b4b45a4dcd7f619446e2" Nov 03 00:43:05.159578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:05.159551 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5"] Nov 03 00:43:10.159875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.159843 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" podUID="f93604d2-5c44-405c-b07c-5dff7d3023dc" containerName="payload-processing" containerID="cri-o://87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0" gracePeriod=25 Nov 03 00:43:10.505069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.505045 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:43:10.574792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.574758 5144 generic.go:356] "Generic (PLEG): container finished" podID="f93604d2-5c44-405c-b07c-5dff7d3023dc" containerID="87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0" exitCode=0 Nov 03 00:43:10.574963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.574805 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" event={"ID":"f93604d2-5c44-405c-b07c-5dff7d3023dc","Type":"ContainerDied","Data":"87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0"} Nov 03 00:43:10.574963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.574820 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" Nov 03 00:43:10.574963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.574877 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5" event={"ID":"f93604d2-5c44-405c-b07c-5dff7d3023dc","Type":"ContainerDied","Data":"a1038b7154088e11e443f432718bffcc4ce063461355f1fff853efe3889eac01"} Nov 03 00:43:10.574963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.574910 5144 scope.go:122] "RemoveContainer" containerID="87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0" Nov 03 00:43:10.585603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.585579 5144 scope.go:122] "RemoveContainer" containerID="87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0" Nov 03 00:43:10.585903 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:43:10.585886 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0\": container with ID starting with 87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0 not found: ID does not exist" containerID="87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0" Nov 03 00:43:10.585960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.585910 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0"} err="failed to get container status \"87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0\": rpc error: code = NotFound desc = could not find container \"87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0\": container with ID starting with 87a1785e61b1b5977a7f2323c14382f52738a034916d51aa7b2685f2c22770e0 not found: ID does not exist" Nov 03 00:43:10.690250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.690212 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg\" (UniqueName: \"kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg\") pod \"f93604d2-5c44-405c-b07c-5dff7d3023dc\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " Nov 03 00:43:10.690250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.690262 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume\") pod \"f93604d2-5c44-405c-b07c-5dff7d3023dc\" (UID: \"f93604d2-5c44-405c-b07c-5dff7d3023dc\") " Nov 03 00:43:10.690677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.690656 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume" pod "f93604d2-5c44-405c-b07c-5dff7d3023dc" (UID: "f93604d2-5c44-405c-b07c-5dff7d3023dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:10.692527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.692504 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg" pod "f93604d2-5c44-405c-b07c-5dff7d3023dc" (UID: "f93604d2-5c44-405c-b07c-5dff7d3023dc"). InnerVolumeSpecName "kube-api-access-khgcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:10.751223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.751153 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx"] Nov 03 00:43:10.751786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.751768 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f93604d2-5c44-405c-b07c-5dff7d3023dc" containerName="payload-processing" Nov 03 00:43:10.751880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.751788 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f93604d2-5c44-405c-b07c-5dff7d3023dc" containerName="payload-processing" Nov 03 00:43:10.751880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.751808 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ec55f256-9d9d-4232-a41e-f61e0aea18df" containerName="revoke-keys" Nov 03 00:43:10.751880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.751815 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec55f256-9d9d-4232-a41e-f61e0aea18df" containerName="revoke-keys" Nov 03 00:43:10.752038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.751991 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec55f256-9d9d-4232-a41e-f61e0aea18df" containerName="revoke-keys" Nov 03 00:43:10.752038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.752019 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f93604d2-5c44-405c-b07c-5dff7d3023dc" containerName="payload-processing" Nov 03 00:43:10.756728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.756708 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.791281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.791255 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.791418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.791398 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kdvj\" (UniqueName: \"kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.791487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.791475 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.791549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.791536 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khgcg\" (UniqueName: \"kubernetes.io/projected/f93604d2-5c44-405c-b07c-5dff7d3023dc-kube-api-access-khgcg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:10.791601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.791555 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f93604d2-5c44-405c-b07c-5dff7d3023dc-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:10.842655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.839901 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:43:10.842655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.839977 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:10.842655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.841538 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx"] Nov 03 00:43:10.892255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.892229 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kdvj\" (UniqueName: \"kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.892390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.892271 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.892390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.892364 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.892917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.892898 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.895214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.895191 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:10.949403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:10.949373 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kdvj\" (UniqueName: \"kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj\") pod \"maas-api-revoke-sub-047a1eda-n69sx\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:11.007391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.007360 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5"] Nov 03 00:43:11.013694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.013661 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ff10d6-758f9788cc-8f6h5"] Nov 03 00:43:11.074989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.074962 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:11.454169 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.454139 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx"] Nov 03 00:43:11.455773 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:11.455733 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e7112cc_f328_43e6_9436_4a0f84a8e055.slice/crio-559900c5261dc8c879a68323e6d488e1cdf8f5ce0df30ab89dfd9c4be163fe4b WatchSource:0}: Error finding container 559900c5261dc8c879a68323e6d488e1cdf8f5ce0df30ab89dfd9c4be163fe4b: Status 404 returned error can't find the container with id 559900c5261dc8c879a68323e6d488e1cdf8f5ce0df30ab89dfd9c4be163fe4b Nov 03 00:43:11.486821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.486790 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv"] Nov 03 00:43:11.492014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.491996 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.498096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.498072 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cncbs\" (UniqueName: \"kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.498190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.498118 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.498230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.498204 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.546937 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.546913 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv"] Nov 03 00:43:11.584844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.584807 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" event={"ID":"8e7112cc-f328-43e6-9436-4a0f84a8e055","Type":"ContainerStarted","Data":"559900c5261dc8c879a68323e6d488e1cdf8f5ce0df30ab89dfd9c4be163fe4b"} Nov 03 00:43:11.598924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.598898 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cncbs\" (UniqueName: \"kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.599055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.598941 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.599055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.598961 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.599604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.599585 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.601440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.601419 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.655279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.655249 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cncbs\" (UniqueName: \"kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs\") pod \"maas-api-revoke-sub-c7c591fb-d5bkv\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.807188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.807154 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:11.994756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.994671 5144 scope.go:122] "RemoveContainer" containerID="4e16009f275184ec7928c5ec0fff36cd1df5aaeec06faaac220a03ee5e0d0e62" Nov 03 00:43:11.997490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:11.997455 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f93604d2-5c44-405c-b07c-5dff7d3023dc" path="/var/lib/kubelet/pods/f93604d2-5c44-405c-b07c-5dff7d3023dc/volumes" Nov 03 00:43:12.154695 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:43:12.154662 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:43:12.192435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:12.192406 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8791297c_2e96_48b9_97df_276f4089a227.slice/crio-2c63550cfd3a7831e5c977b8d542eb9571b2d16c5e06cd3f4e802a6389c8e456 WatchSource:0}: Error finding container 2c63550cfd3a7831e5c977b8d542eb9571b2d16c5e06cd3f4e802a6389c8e456: Status 404 returned error can't find the container with id 2c63550cfd3a7831e5c977b8d542eb9571b2d16c5e06cd3f4e802a6389c8e456 Nov 03 00:43:12.192610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:12.192587 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv"] Nov 03 00:43:12.594365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:12.594327 5144 generic.go:356] "Generic (PLEG): container finished" podID="8e7112cc-f328-43e6-9436-4a0f84a8e055" containerID="e28bf4287570869977f7f112e8d21294938ea82b09f0ee560098aacfd5a05fe8" exitCode=0 Nov 03 00:43:12.594772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:12.594408 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" event={"ID":"8e7112cc-f328-43e6-9436-4a0f84a8e055","Type":"ContainerDied","Data":"e28bf4287570869977f7f112e8d21294938ea82b09f0ee560098aacfd5a05fe8"} Nov 03 00:43:12.595865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:12.595842 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" event={"ID":"8791297c-2e96-48b9-97df-276f4089a227","Type":"ContainerStarted","Data":"2c63550cfd3a7831e5c977b8d542eb9571b2d16c5e06cd3f4e802a6389c8e456"} Nov 03 00:43:13.603198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.603160 5144 generic.go:356] "Generic (PLEG): container finished" podID="8791297c-2e96-48b9-97df-276f4089a227" containerID="8ec844b4d93ec10ea850cd7a25cf541f46894bf8b60787f3e5e50e20a744cde4" exitCode=0 Nov 03 00:43:13.603679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.603249 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" event={"ID":"8791297c-2e96-48b9-97df-276f4089a227","Type":"ContainerDied","Data":"8ec844b4d93ec10ea850cd7a25cf541f46894bf8b60787f3e5e50e20a744cde4"} Nov 03 00:43:13.762705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.762669 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:13.829748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.829717 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca\") pod \"8e7112cc-f328-43e6-9436-4a0f84a8e055\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " Nov 03 00:43:13.829916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.829808 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj\" (UniqueName: \"kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj\") pod \"8e7112cc-f328-43e6-9436-4a0f84a8e055\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " Nov 03 00:43:13.829916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.829901 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca\") pod \"8e7112cc-f328-43e6-9436-4a0f84a8e055\" (UID: \"8e7112cc-f328-43e6-9436-4a0f84a8e055\") " Nov 03 00:43:13.830135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.830112 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca" pod "8e7112cc-f328-43e6-9436-4a0f84a8e055" (UID: "8e7112cc-f328-43e6-9436-4a0f84a8e055"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:13.830368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.830338 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:13.832212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.832187 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj" pod "8e7112cc-f328-43e6-9436-4a0f84a8e055" (UID: "8e7112cc-f328-43e6-9436-4a0f84a8e055"). InnerVolumeSpecName "kube-api-access-9kdvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:13.832425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.832405 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca" pod "8e7112cc-f328-43e6-9436-4a0f84a8e055" (UID: "8e7112cc-f328-43e6-9436-4a0f84a8e055"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:13.931402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.931323 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8e7112cc-f328-43e6-9436-4a0f84a8e055-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:13.931402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:13.931359 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kdvj\" (UniqueName: \"kubernetes.io/projected/8e7112cc-f328-43e6-9436-4a0f84a8e055-kube-api-access-9kdvj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:14.611967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.611936 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" Nov 03 00:43:14.612379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.611934 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx" event={"ID":"8e7112cc-f328-43e6-9436-4a0f84a8e055","Type":"ContainerDied","Data":"559900c5261dc8c879a68323e6d488e1cdf8f5ce0df30ab89dfd9c4be163fe4b"} Nov 03 00:43:14.612379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.612061 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="559900c5261dc8c879a68323e6d488e1cdf8f5ce0df30ab89dfd9c4be163fe4b" Nov 03 00:43:14.783818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.783790 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:14.841325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.841302 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs\" (UniqueName: \"kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs\") pod \"8791297c-2e96-48b9-97df-276f4089a227\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " Nov 03 00:43:14.841478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.841336 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca\") pod \"8791297c-2e96-48b9-97df-276f4089a227\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " Nov 03 00:43:14.841478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.841381 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca\") pod \"8791297c-2e96-48b9-97df-276f4089a227\" (UID: \"8791297c-2e96-48b9-97df-276f4089a227\") " Nov 03 00:43:14.841780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.841748 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca" pod "8791297c-2e96-48b9-97df-276f4089a227" (UID: "8791297c-2e96-48b9-97df-276f4089a227"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:14.841984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.841966 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8791297c-2e96-48b9-97df-276f4089a227-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:14.844171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.844143 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca" pod "8791297c-2e96-48b9-97df-276f4089a227" (UID: "8791297c-2e96-48b9-97df-276f4089a227"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:14.844255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.844166 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs" pod "8791297c-2e96-48b9-97df-276f4089a227" (UID: "8791297c-2e96-48b9-97df-276f4089a227"). InnerVolumeSpecName "kube-api-access-cncbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:14.942955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.942890 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8791297c-2e96-48b9-97df-276f4089a227-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:14.942955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:14.942917 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cncbs\" (UniqueName: \"kubernetes.io/projected/8791297c-2e96-48b9-97df-276f4089a227-kube-api-access-cncbs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:15.621131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:15.621100 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" event={"ID":"8791297c-2e96-48b9-97df-276f4089a227","Type":"ContainerDied","Data":"2c63550cfd3a7831e5c977b8d542eb9571b2d16c5e06cd3f4e802a6389c8e456"} Nov 03 00:43:15.621505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:15.621151 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c63550cfd3a7831e5c977b8d542eb9571b2d16c5e06cd3f4e802a6389c8e456" Nov 03 00:43:15.621505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:15.621112 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv" Nov 03 00:43:27.631039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:27.630999 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s"] Nov 03 00:43:27.632828 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:27.632786 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" containerID="cri-o://892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175" gracePeriod=30 Nov 03 00:43:32.920212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:32.920181 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:43:33.044676 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.044652 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.044814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.044693 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.044814 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.044796 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.044910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.044838 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.044910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.044854 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:33.044910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.044876 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.045077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045014 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.045077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045044 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.045077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045073 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.045250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045085 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:33.045250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045106 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv\") pod \"a882e75d-edd8-459a-9c51-d2834907464a\" (UID: \"a882e75d-edd8-459a-9c51-d2834907464a\") " Nov 03 00:43:33.045359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045251 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:33.045454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045431 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:33.045454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045429 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:33.045786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045767 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.045875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045792 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.045875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045806 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a882e75d-edd8-459a-9c51-d2834907464a-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.045875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045821 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.045875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.045836 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.047075 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.047053 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:43:33.047257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.047236 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:43:33.047588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.047566 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:33.047666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.047601 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv" pod "a882e75d-edd8-459a-9c51-d2834907464a" (UID: "a882e75d-edd8-459a-9c51-d2834907464a"). InnerVolumeSpecName "kube-api-access-75bxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:33.147046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.147019 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75bxv\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-kube-api-access-75bxv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.147046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.147043 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a882e75d-edd8-459a-9c51-d2834907464a-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.147222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.147052 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a882e75d-edd8-459a-9c51-d2834907464a-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.147222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.147062 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a882e75d-edd8-459a-9c51-d2834907464a-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:33.757565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.757526 5144 generic.go:356] "Generic (PLEG): container finished" podID="a882e75d-edd8-459a-9c51-d2834907464a" containerID="892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175" exitCode=0 Nov 03 00:43:33.757565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.757566 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" event={"ID":"a882e75d-edd8-459a-9c51-d2834907464a","Type":"ContainerDied","Data":"892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175"} Nov 03 00:43:33.757864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.757588 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" Nov 03 00:43:33.757864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.757613 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s" event={"ID":"a882e75d-edd8-459a-9c51-d2834907464a","Type":"ContainerDied","Data":"1dbfdf3b6156aab191580b2b77e259977688e8ad5c0134ef941eb126c84393e6"} Nov 03 00:43:33.757864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.757650 5144 scope.go:122] "RemoveContainer" containerID="892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175" Nov 03 00:43:33.769858 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.769839 5144 scope.go:122] "RemoveContainer" containerID="f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204" Nov 03 00:43:33.782044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.782016 5144 scope.go:122] "RemoveContainer" containerID="892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175" Nov 03 00:43:33.782336 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:43:33.782318 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175\": container with ID starting with 892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175 not found: ID does not exist" containerID="892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175" Nov 03 00:43:33.782420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.782348 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175"} err="failed to get container status \"892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175\": rpc error: code = NotFound desc = could not find container \"892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175\": container with ID starting with 892d0bfc90c8402c2ec05ffc7e5b7f51d95f787c5d6037b5a03fca28c90e0175 not found: ID does not exist" Nov 03 00:43:33.782420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.782381 5144 scope.go:122] "RemoveContainer" containerID="f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204" Nov 03 00:43:33.782684 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:43:33.782662 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204\": container with ID starting with f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204 not found: ID does not exist" containerID="f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204" Nov 03 00:43:33.782740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.782695 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204"} err="failed to get container status \"f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204\": rpc error: code = NotFound desc = could not find container \"f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204\": container with ID starting with f6dfeaf03b3b56eed8223f09f9d462c4360a79bbdbac85feaa4a7798c2ecf204 not found: ID does not exist" Nov 03 00:43:33.974438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:33.974401 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s"] Nov 03 00:43:34.026709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.026610 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-ff10d6-openshift-default-974bcddd5-8np7s"] Nov 03 00:43:34.048149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.048120 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469"] Nov 03 00:43:34.048979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.048960 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8e7112cc-f328-43e6-9436-4a0f84a8e055" containerName="revoke-keys" Nov 03 00:43:34.049020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.048986 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e7112cc-f328-43e6-9436-4a0f84a8e055" containerName="revoke-keys" Nov 03 00:43:34.049020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049003 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8791297c-2e96-48b9-97df-276f4089a227" containerName="revoke-keys" Nov 03 00:43:34.049020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049011 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8791297c-2e96-48b9-97df-276f4089a227" containerName="revoke-keys" Nov 03 00:43:34.049124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049026 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" Nov 03 00:43:34.049124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049035 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" Nov 03 00:43:34.049124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049067 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" Nov 03 00:43:34.049124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049075 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" Nov 03 00:43:34.049244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049233 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e7112cc-f328-43e6-9436-4a0f84a8e055" containerName="revoke-keys" Nov 03 00:43:34.049277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049248 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8791297c-2e96-48b9-97df-276f4089a227" containerName="revoke-keys" Nov 03 00:43:34.049277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049258 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" Nov 03 00:43:34.049277 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.049270 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a882e75d-edd8-459a-9c51-d2834907464a" containerName="istio-proxy" Nov 03 00:43:34.054722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.054703 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.106266 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.106240 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469"] Nov 03 00:43:34.106508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.106491 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:34.106709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.106691 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:43:34.157463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.157422 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.157598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.157498 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmwsr\" (UniqueName: \"kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.157598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.157529 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.259240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.259204 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.259402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.259302 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmwsr\" (UniqueName: \"kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.259402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.259336 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.260083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.260050 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.261898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.261874 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.315795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.315728 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmwsr\" (UniqueName: \"kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr\") pod \"maas-api-revoke-sub-667efdf8-4j469\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.373162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.373124 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:34.752833 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:34.752799 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8bb83b0_2b44_4559_b9ab_97924f9c8f0f.slice/crio-8ce7d36d508c175aadb97c44212033e0a449ee83eaddc01c666d49a3fab4e910 WatchSource:0}: Error finding container 8ce7d36d508c175aadb97c44212033e0a449ee83eaddc01c666d49a3fab4e910: Status 404 returned error can't find the container with id 8ce7d36d508c175aadb97c44212033e0a449ee83eaddc01c666d49a3fab4e910 Nov 03 00:43:34.754055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.754030 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469"] Nov 03 00:43:34.787369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:34.787325 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" event={"ID":"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f","Type":"ContainerStarted","Data":"8ce7d36d508c175aadb97c44212033e0a449ee83eaddc01c666d49a3fab4e910"} Nov 03 00:43:35.798708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:35.798679 5144 generic.go:356] "Generic (PLEG): container finished" podID="b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" containerID="d89870d1b2fd6c57df18a5781f000c56719eeb86979d45babfa00f5b55b849bf" exitCode=0 Nov 03 00:43:35.799077 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:35.798767 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" event={"ID":"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f","Type":"ContainerDied","Data":"d89870d1b2fd6c57df18a5781f000c56719eeb86979d45babfa00f5b55b849bf"} Nov 03 00:43:35.997329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:35.997260 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a882e75d-edd8-459a-9c51-d2834907464a" path="/var/lib/kubelet/pods/a882e75d-edd8-459a-9c51-d2834907464a/volumes" Nov 03 00:43:36.974871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:36.974844 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:37.092197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.092106 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr\" (UniqueName: \"kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr\") pod \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " Nov 03 00:43:37.092197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.092159 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca\") pod \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " Nov 03 00:43:37.092491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.092214 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca\") pod \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\" (UID: \"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f\") " Nov 03 00:43:37.092664 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.092623 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca" pod "b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" (UID: "b8bb83b0-2b44-4559-b9ab-97924f9c8f0f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:37.092890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.092778 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:37.094913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.094886 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca" pod "b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" (UID: "b8bb83b0-2b44-4559-b9ab-97924f9c8f0f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:37.094913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.094894 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr" pod "b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" (UID: "b8bb83b0-2b44-4559-b9ab-97924f9c8f0f"). InnerVolumeSpecName "kube-api-access-rmwsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:37.194424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.194395 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmwsr\" (UniqueName: \"kubernetes.io/projected/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-kube-api-access-rmwsr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:37.194424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.194422 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:37.814454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.814419 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" event={"ID":"b8bb83b0-2b44-4559-b9ab-97924f9c8f0f","Type":"ContainerDied","Data":"8ce7d36d508c175aadb97c44212033e0a449ee83eaddc01c666d49a3fab4e910"} Nov 03 00:43:37.814454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.814452 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469" Nov 03 00:43:37.814667 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:37.814478 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ce7d36d508c175aadb97c44212033e0a449ee83eaddc01c666d49a3fab4e910" Nov 03 00:43:40.193653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.193604 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb"] Nov 03 00:43:40.194440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.194417 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" containerName="revoke-keys" Nov 03 00:43:40.194440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.194441 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" containerName="revoke-keys" Nov 03 00:43:40.194603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.194591 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" containerName="revoke-keys" Nov 03 00:43:40.198121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.198100 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.222286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.222260 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.222406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.222298 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.222406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.222323 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq8x8\" (UniqueName: \"kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.252990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.252963 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb"] Nov 03 00:43:40.298043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.298015 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:43:40.298186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.298016 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:40.323123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.323099 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.323252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.323138 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.323252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.323167 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq8x8\" (UniqueName: \"kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.323943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.323911 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.325859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.325837 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.358594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.358568 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d"] Nov 03 00:43:40.362851 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.362836 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.381043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.381022 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq8x8\" (UniqueName: \"kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8\") pod \"maas-api-revoke-sub-0cd2322f-m7fdb\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.419650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.419603 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d"] Nov 03 00:43:40.424236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.424206 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.424390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.424331 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.424390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.424370 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsc9x\" (UniqueName: \"kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.511334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.511306 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:40.535573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.532424 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.535573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.532619 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.535573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.532711 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsc9x\" (UniqueName: \"kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.535573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.533947 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.538568 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.536529 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.591653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.589887 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsc9x\" (UniqueName: \"kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x\") pod \"maas-api-revoke-sub-022f7cf2-kv92d\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.676913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.676882 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:40.938011 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:40.937982 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2ed77bd_0674_4f6f_92d7_d61592b2bbb5.slice/crio-724e810b23b968b633de7bbd07225ed7272fcd850a0ca6ce5cedbeb5f7c909a7 WatchSource:0}: Error finding container 724e810b23b968b633de7bbd07225ed7272fcd850a0ca6ce5cedbeb5f7c909a7: Status 404 returned error can't find the container with id 724e810b23b968b633de7bbd07225ed7272fcd850a0ca6ce5cedbeb5f7c909a7 Nov 03 00:43:40.938973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:40.938942 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb"] Nov 03 00:43:41.057824 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:41.057781 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25698ac1_1c6b_40d4_8cdc_349692af65be.slice/crio-93c94018addd6a85742686367d3522448797ffa75501e6b2295e1c7a9a405820 WatchSource:0}: Error finding container 93c94018addd6a85742686367d3522448797ffa75501e6b2295e1c7a9a405820: Status 404 returned error can't find the container with id 93c94018addd6a85742686367d3522448797ffa75501e6b2295e1c7a9a405820 Nov 03 00:43:41.059533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:41.059504 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d"] Nov 03 00:43:41.843923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:41.843886 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" event={"ID":"25698ac1-1c6b-40d4-8cdc-349692af65be","Type":"ContainerStarted","Data":"98c85584fad5f7ed45e6a4fbdf7710e8455ad1705926397f42004f91f40df434"} Nov 03 00:43:41.844393 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:41.843957 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" event={"ID":"25698ac1-1c6b-40d4-8cdc-349692af65be","Type":"ContainerStarted","Data":"93c94018addd6a85742686367d3522448797ffa75501e6b2295e1c7a9a405820"} Nov 03 00:43:41.845880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:41.845849 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" event={"ID":"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5","Type":"ContainerStarted","Data":"5ab382cbd80a45717d65e4b19153843bae5d5baebd1ac2c6020332787ec8c6e6"} Nov 03 00:43:41.845997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:41.845905 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" event={"ID":"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5","Type":"ContainerStarted","Data":"724e810b23b968b633de7bbd07225ed7272fcd850a0ca6ce5cedbeb5f7c909a7"} Nov 03 00:43:41.957951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:41.957903 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" podStartSLOduration=1.9578890740000001 podStartE2EDuration="1.957889074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:41.955436088 +0000 UTC m=+2312.101130566" watchObservedRunningTime="2026-11-03 00:43:41.957889074 +0000 UTC m=+2312.103583529" Nov 03 00:43:42.069462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:42.069404 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" podStartSLOduration=2.069390884 podStartE2EDuration="2.069390884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:42.066169488 +0000 UTC m=+2312.211863954" watchObservedRunningTime="2026-11-03 00:43:42.069390884 +0000 UTC m=+2312.215085367" Nov 03 00:43:42.155055 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:43:42.154973 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:43:42.855198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:42.855165 5144 generic.go:356] "Generic (PLEG): container finished" podID="25698ac1-1c6b-40d4-8cdc-349692af65be" containerID="98c85584fad5f7ed45e6a4fbdf7710e8455ad1705926397f42004f91f40df434" exitCode=0 Nov 03 00:43:42.855706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:42.855246 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" event={"ID":"25698ac1-1c6b-40d4-8cdc-349692af65be","Type":"ContainerDied","Data":"98c85584fad5f7ed45e6a4fbdf7710e8455ad1705926397f42004f91f40df434"} Nov 03 00:43:42.857004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:42.856976 5144 generic.go:356] "Generic (PLEG): container finished" podID="b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" containerID="5ab382cbd80a45717d65e4b19153843bae5d5baebd1ac2c6020332787ec8c6e6" exitCode=0 Nov 03 00:43:42.857140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:42.857058 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" event={"ID":"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5","Type":"ContainerDied","Data":"5ab382cbd80a45717d65e4b19153843bae5d5baebd1ac2c6020332787ec8c6e6"} Nov 03 00:43:44.021467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.021443 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:44.060028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.060002 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:44.072333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.072311 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8\" (UniqueName: \"kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8\") pod \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " Nov 03 00:43:44.072429 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.072344 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca\") pod \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " Nov 03 00:43:44.072429 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.072392 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca\") pod \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\" (UID: \"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5\") " Nov 03 00:43:44.072839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.072809 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca" pod "b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" (UID: "b2ed77bd-0674-4f6f-92d7-d61592b2bbb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:44.072947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.072932 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:44.074582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.074561 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8" pod "b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" (UID: "b2ed77bd-0674-4f6f-92d7-d61592b2bbb5"). InnerVolumeSpecName "kube-api-access-kq8x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:44.074684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.074607 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca" pod "b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" (UID: "b2ed77bd-0674-4f6f-92d7-d61592b2bbb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:44.174102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.174038 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca\") pod \"25698ac1-1c6b-40d4-8cdc-349692af65be\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " Nov 03 00:43:44.174233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.174129 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x\" (UniqueName: \"kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x\") pod \"25698ac1-1c6b-40d4-8cdc-349692af65be\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " Nov 03 00:43:44.174233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.174148 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca\") pod \"25698ac1-1c6b-40d4-8cdc-349692af65be\" (UID: \"25698ac1-1c6b-40d4-8cdc-349692af65be\") " Nov 03 00:43:44.174435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.174412 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca" pod "25698ac1-1c6b-40d4-8cdc-349692af65be" (UID: "25698ac1-1c6b-40d4-8cdc-349692af65be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:44.174488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.174421 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq8x8\" (UniqueName: \"kubernetes.io/projected/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-kube-api-access-kq8x8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:44.174488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.174469 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:44.176287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.176267 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x" pod "25698ac1-1c6b-40d4-8cdc-349692af65be" (UID: "25698ac1-1c6b-40d4-8cdc-349692af65be"). InnerVolumeSpecName "kube-api-access-wsc9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:44.176363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.176329 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca" pod "25698ac1-1c6b-40d4-8cdc-349692af65be" (UID: "25698ac1-1c6b-40d4-8cdc-349692af65be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:44.275623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.275588 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsc9x\" (UniqueName: \"kubernetes.io/projected/25698ac1-1c6b-40d4-8cdc-349692af65be-kube-api-access-wsc9x\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:44.275833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.275624 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:44.275833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.275659 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25698ac1-1c6b-40d4-8cdc-349692af65be-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:44.871719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.871674 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" event={"ID":"b2ed77bd-0674-4f6f-92d7-d61592b2bbb5","Type":"ContainerDied","Data":"724e810b23b968b633de7bbd07225ed7272fcd850a0ca6ce5cedbeb5f7c909a7"} Nov 03 00:43:44.871921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.871774 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="724e810b23b968b633de7bbd07225ed7272fcd850a0ca6ce5cedbeb5f7c909a7" Nov 03 00:43:44.871921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.871689 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb" Nov 03 00:43:44.873253 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.873233 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" Nov 03 00:43:44.873395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.873235 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d" event={"ID":"25698ac1-1c6b-40d4-8cdc-349692af65be","Type":"ContainerDied","Data":"93c94018addd6a85742686367d3522448797ffa75501e6b2295e1c7a9a405820"} Nov 03 00:43:44.873395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:44.873337 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="93c94018addd6a85742686367d3522448797ffa75501e6b2295e1c7a9a405820" Nov 03 00:43:55.457781 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.457747 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp"] Nov 03 00:43:55.458598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.458570 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" containerName="revoke-keys" Nov 03 00:43:55.458598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.458595 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" containerName="revoke-keys" Nov 03 00:43:55.458813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.458677 5144 cpu_manager.go:414] "containerMap: removing container" podUID="25698ac1-1c6b-40d4-8cdc-349692af65be" containerName="revoke-keys" Nov 03 00:43:55.458813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.458686 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25698ac1-1c6b-40d4-8cdc-349692af65be" containerName="revoke-keys" Nov 03 00:43:55.458884 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.458824 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="25698ac1-1c6b-40d4-8cdc-349692af65be" containerName="revoke-keys" Nov 03 00:43:55.458884 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.458840 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" containerName="revoke-keys" Nov 03 00:43:55.464049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.464029 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.520559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.520529 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp"] Nov 03 00:43:55.560846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.560813 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:43:55.561004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.560885 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:43:55.591598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.591571 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.591809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.591788 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzl8p\" (UniqueName: \"kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.591864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.591822 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.693043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.693016 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.693217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.693199 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzl8p\" (UniqueName: \"kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.693298 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.693235 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.693834 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.693811 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.695340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.695322 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.749325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.749266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzl8p\" (UniqueName: \"kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p\") pod \"maas-api-revoke-sub-5131c990-kwlhp\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.778442 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.778424 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:55.868198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.868171 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6"] Nov 03 00:43:55.873339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.873319 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:55.926964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.926939 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6"] Nov 03 00:43:55.996518 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.996475 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmth9\" (UniqueName: \"kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:55.996706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.996537 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:55.996706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:55.996639 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.097798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.097768 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.097961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.097889 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmth9\" (UniqueName: \"kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.097961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.097928 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.098443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.098417 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.100319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.100291 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.153830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.153809 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmth9\" (UniqueName: \"kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9\") pod \"maas-api-revoke-sub-faaece96-jrxp6\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.166512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.166490 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp"] Nov 03 00:43:56.166681 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:56.166660 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a968356_24f7_45de_9916_bc4d2769d1c6.slice/crio-532b7ab3cc2f4ed3766e7254bd6b1c11b9d4a2851602eea9709ec68cba485d52 WatchSource:0}: Error finding container 532b7ab3cc2f4ed3766e7254bd6b1c11b9d4a2851602eea9709ec68cba485d52: Status 404 returned error can't find the container with id 532b7ab3cc2f4ed3766e7254bd6b1c11b9d4a2851602eea9709ec68cba485d52 Nov 03 00:43:56.191410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.191390 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:56.564835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.564807 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6"] Nov 03 00:43:56.565607 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:43:56.565582 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1a2756_8359_431d_8c11_c00d8504fdd1.slice/crio-d6c91d39088ebec1a20b3b5dc46a0bd9242e2fdbaa76f631052466e202a7b049 WatchSource:0}: Error finding container d6c91d39088ebec1a20b3b5dc46a0bd9242e2fdbaa76f631052466e202a7b049: Status 404 returned error can't find the container with id d6c91d39088ebec1a20b3b5dc46a0bd9242e2fdbaa76f631052466e202a7b049 Nov 03 00:43:56.970285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.970254 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" event={"ID":"0a1a2756-8359-431d-8c11-c00d8504fdd1","Type":"ContainerStarted","Data":"84a0e71f371816736b533de9fee2b234be9b579b5c8d65693445b827ab0e83ee"} Nov 03 00:43:56.970464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.970308 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" event={"ID":"0a1a2756-8359-431d-8c11-c00d8504fdd1","Type":"ContainerStarted","Data":"d6c91d39088ebec1a20b3b5dc46a0bd9242e2fdbaa76f631052466e202a7b049"} Nov 03 00:43:56.971928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.971901 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" event={"ID":"1a968356-24f7-45de-9916-bc4d2769d1c6","Type":"ContainerStarted","Data":"2a4c68ef86da8938d18e189559fe799f655ff9a114bcf82607056d9682f5c2f9"} Nov 03 00:43:56.972027 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:56.971954 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" event={"ID":"1a968356-24f7-45de-9916-bc4d2769d1c6","Type":"ContainerStarted","Data":"532b7ab3cc2f4ed3766e7254bd6b1c11b9d4a2851602eea9709ec68cba485d52"} Nov 03 00:43:57.085192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:57.085143 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" podStartSLOduration=2.085130023 podStartE2EDuration="2.085130023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:57.081488144 +0000 UTC m=+2327.227182618" watchObservedRunningTime="2026-11-03 00:43:57.085130023 +0000 UTC m=+2327.230824477" Nov 03 00:43:57.199092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:57.195713 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" podStartSLOduration=2.195696232 podStartE2EDuration="2.195696232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:57.193690318 +0000 UTC m=+2327.339384799" watchObservedRunningTime="2026-11-03 00:43:57.195696232 +0000 UTC m=+2327.341390687" Nov 03 00:43:57.981324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:57.981245 5144 generic.go:356] "Generic (PLEG): container finished" podID="0a1a2756-8359-431d-8c11-c00d8504fdd1" containerID="84a0e71f371816736b533de9fee2b234be9b579b5c8d65693445b827ab0e83ee" exitCode=0 Nov 03 00:43:57.981771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:57.981326 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" event={"ID":"0a1a2756-8359-431d-8c11-c00d8504fdd1","Type":"ContainerDied","Data":"84a0e71f371816736b533de9fee2b234be9b579b5c8d65693445b827ab0e83ee"} Nov 03 00:43:57.982894 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:57.982875 5144 generic.go:356] "Generic (PLEG): container finished" podID="1a968356-24f7-45de-9916-bc4d2769d1c6" containerID="2a4c68ef86da8938d18e189559fe799f655ff9a114bcf82607056d9682f5c2f9" exitCode=0 Nov 03 00:43:57.983018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:57.982900 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" event={"ID":"1a968356-24f7-45de-9916-bc4d2769d1c6","Type":"ContainerDied","Data":"2a4c68ef86da8938d18e189559fe799f655ff9a114bcf82607056d9682f5c2f9"} Nov 03 00:43:59.150839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.150201 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:43:59.175957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.175936 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:43:59.233378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.233354 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca\") pod \"1a968356-24f7-45de-9916-bc4d2769d1c6\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " Nov 03 00:43:59.233520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.233406 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p\" (UniqueName: \"kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p\") pod \"1a968356-24f7-45de-9916-bc4d2769d1c6\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " Nov 03 00:43:59.233576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.233522 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca\") pod \"1a968356-24f7-45de-9916-bc4d2769d1c6\" (UID: \"1a968356-24f7-45de-9916-bc4d2769d1c6\") " Nov 03 00:43:59.233802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.233778 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca" pod "1a968356-24f7-45de-9916-bc4d2769d1c6" (UID: "1a968356-24f7-45de-9916-bc4d2769d1c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:59.234161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.234105 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:59.235733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.235710 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca" pod "1a968356-24f7-45de-9916-bc4d2769d1c6" (UID: "1a968356-24f7-45de-9916-bc4d2769d1c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:59.236081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.236063 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p" pod "1a968356-24f7-45de-9916-bc4d2769d1c6" (UID: "1a968356-24f7-45de-9916-bc4d2769d1c6"). InnerVolumeSpecName "kube-api-access-fzl8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:59.335447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.335415 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca\") pod \"0a1a2756-8359-431d-8c11-c00d8504fdd1\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " Nov 03 00:43:59.335591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.335514 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca\") pod \"0a1a2756-8359-431d-8c11-c00d8504fdd1\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " Nov 03 00:43:59.335679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.335662 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9\" (UniqueName: \"kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9\") pod \"0a1a2756-8359-431d-8c11-c00d8504fdd1\" (UID: \"0a1a2756-8359-431d-8c11-c00d8504fdd1\") " Nov 03 00:43:59.335859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.335839 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca" pod "0a1a2756-8359-431d-8c11-c00d8504fdd1" (UID: "0a1a2756-8359-431d-8c11-c00d8504fdd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:43:59.336067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.336044 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzl8p\" (UniqueName: \"kubernetes.io/projected/1a968356-24f7-45de-9916-bc4d2769d1c6-kube-api-access-fzl8p\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:59.336067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.336066 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a968356-24f7-45de-9916-bc4d2769d1c6-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:59.336230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.336080 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:59.337798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.337776 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca" pod "0a1a2756-8359-431d-8c11-c00d8504fdd1" (UID: "0a1a2756-8359-431d-8c11-c00d8504fdd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:43:59.337879 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.337797 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9" pod "0a1a2756-8359-431d-8c11-c00d8504fdd1" (UID: "0a1a2756-8359-431d-8c11-c00d8504fdd1"). InnerVolumeSpecName "kube-api-access-wmth9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:43:59.437733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.437699 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a1a2756-8359-431d-8c11-c00d8504fdd1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:43:59.437733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:43:59.437734 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmth9\" (UniqueName: \"kubernetes.io/projected/0a1a2756-8359-431d-8c11-c00d8504fdd1-kube-api-access-wmth9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:00.001403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:00.001363 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" Nov 03 00:44:00.001403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:00.001430 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp" event={"ID":"1a968356-24f7-45de-9916-bc4d2769d1c6","Type":"ContainerDied","Data":"532b7ab3cc2f4ed3766e7254bd6b1c11b9d4a2851602eea9709ec68cba485d52"} Nov 03 00:44:00.001840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:00.001474 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532b7ab3cc2f4ed3766e7254bd6b1c11b9d4a2851602eea9709ec68cba485d52" Nov 03 00:44:00.003285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:00.003254 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" event={"ID":"0a1a2756-8359-431d-8c11-c00d8504fdd1","Type":"ContainerDied","Data":"d6c91d39088ebec1a20b3b5dc46a0bd9242e2fdbaa76f631052466e202a7b049"} Nov 03 00:44:00.003416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:00.003299 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6c91d39088ebec1a20b3b5dc46a0bd9242e2fdbaa76f631052466e202a7b049" Nov 03 00:44:00.003416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:00.003351 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6" Nov 03 00:44:02.772415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.772376 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d"] Nov 03 00:44:02.773165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.773142 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0a1a2756-8359-431d-8c11-c00d8504fdd1" containerName="revoke-keys" Nov 03 00:44:02.773270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.773167 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1a2756-8359-431d-8c11-c00d8504fdd1" containerName="revoke-keys" Nov 03 00:44:02.773270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.773181 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1a968356-24f7-45de-9916-bc4d2769d1c6" containerName="revoke-keys" Nov 03 00:44:02.773270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.773189 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a968356-24f7-45de-9916-bc4d2769d1c6" containerName="revoke-keys" Nov 03 00:44:02.773422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.773386 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1a2756-8359-431d-8c11-c00d8504fdd1" containerName="revoke-keys" Nov 03 00:44:02.773422 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.773400 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a968356-24f7-45de-9916-bc4d2769d1c6" containerName="revoke-keys" Nov 03 00:44:02.777144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.777123 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878461 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878538 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878573 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878609 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdddh\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878681 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878651 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878692 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878773 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878799 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.878909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.878839 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.885198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.885174 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d"] Nov 03 00:44:02.979875 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.979845 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.979904 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.979956 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.979981 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980018 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdddh\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980044 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980085 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980172 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980202 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980360 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980390 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980480 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980501 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.980813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.980789 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.982802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.982770 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:02.982938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:02.982913 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:03.038389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.038318 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:03.038510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.038444 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdddh\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:03.091250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.091224 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:03.527866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.527824 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d"] Nov 03 00:44:03.533360 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:03.533325 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3774f1b2_87c9_4fb4_b67a_7cc7c808a60e.slice/crio-052294b6f42e07fbf1756b02e8b9491d48de11a2da527f459de2570fefb111f8 WatchSource:0}: Error finding container 052294b6f42e07fbf1756b02e8b9491d48de11a2da527f459de2570fefb111f8: Status 404 returned error can't find the container with id 052294b6f42e07fbf1756b02e8b9491d48de11a2da527f459de2570fefb111f8 Nov 03 00:44:03.536137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.536100 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:03.536221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.536187 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:03.536267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:03.536229 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:04.035557 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:04.035521 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" event={"ID":"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e","Type":"ContainerStarted","Data":"7f495125f2af637428c61eead0d3a780ac427b6238916e981fedfad71e0dc22b"} Nov 03 00:44:04.035986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:04.035582 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" event={"ID":"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e","Type":"ContainerStarted","Data":"052294b6f42e07fbf1756b02e8b9491d48de11a2da527f459de2570fefb111f8"} Nov 03 00:44:04.091793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:04.091769 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:04.093398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:04.093351 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:04.093542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:04.093434 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:04.155990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:04.155943 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podStartSLOduration=2.155930503 podStartE2EDuration="2.155930503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:04.151877117 +0000 UTC m=+2334.297571574" watchObservedRunningTime="2026-11-03 00:44:04.155930503 +0000 UTC m=+2334.301625026" Nov 03 00:44:05.091738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.091703 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:05.092133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.091762 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:05.244777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.244736 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4"] Nov 03 00:44:05.250829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.250800 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.290859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.290828 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4"] Nov 03 00:44:05.344986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.344891 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:44:05.344986 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.344891 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:05.409708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.409680 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.409871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.409727 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vsn8\" (UniqueName: \"kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.409871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.409831 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.511216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.511171 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.511408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.511246 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vsn8\" (UniqueName: \"kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.511408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.511303 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.511886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.511858 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.513966 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.513939 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.567650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.567604 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vsn8\" (UniqueName: \"kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8\") pod \"maas-api-revoke-sub-4c120a8d-w78f4\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:05.865296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:05.865257 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:06.091657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:06.091596 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:06.091840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:06.091688 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:06.270766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:06.270731 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4"] Nov 03 00:44:06.271222 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:06.271201 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5745d317_84cc_4bf2_9a4d_1017245c2cbf.slice/crio-55661872ef8dfa21f48f3b53e0bccd3863759dbe401b73229e76ab83ff838543 WatchSource:0}: Error finding container 55661872ef8dfa21f48f3b53e0bccd3863759dbe401b73229e76ab83ff838543: Status 404 returned error can't find the container with id 55661872ef8dfa21f48f3b53e0bccd3863759dbe401b73229e76ab83ff838543 Nov 03 00:44:07.072286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:07.072252 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" event={"ID":"5745d317-84cc-4bf2-9a4d-1017245c2cbf","Type":"ContainerStarted","Data":"7d3b17bd27a9fac25ede8b5a06a62770b5e51e7685cb1417c7c296594eee7c84"} Nov 03 00:44:07.072481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:07.072299 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" event={"ID":"5745d317-84cc-4bf2-9a4d-1017245c2cbf","Type":"ContainerStarted","Data":"55661872ef8dfa21f48f3b53e0bccd3863759dbe401b73229e76ab83ff838543"} Nov 03 00:44:07.091576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:07.091545 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:07.091749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:07.091590 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:07.190228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:07.190181 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" podStartSLOduration=2.1901671719999998 podStartE2EDuration="2.190167172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:07.188031136 +0000 UTC m=+2337.333725613" watchObservedRunningTime="2026-11-03 00:44:07.190167172 +0000 UTC m=+2337.335861642" Nov 03 00:44:08.081095 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:08.081062 5144 generic.go:356] "Generic (PLEG): container finished" podID="5745d317-84cc-4bf2-9a4d-1017245c2cbf" containerID="7d3b17bd27a9fac25ede8b5a06a62770b5e51e7685cb1417c7c296594eee7c84" exitCode=0 Nov 03 00:44:08.081250 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:08.081131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" event={"ID":"5745d317-84cc-4bf2-9a4d-1017245c2cbf","Type":"ContainerDied","Data":"7d3b17bd27a9fac25ede8b5a06a62770b5e51e7685cb1417c7c296594eee7c84"} Nov 03 00:44:08.091877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:08.091848 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:08.092081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:08.091900 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:09.091726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.091688 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:09.092105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.091740 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:09.278360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.278337 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:09.389062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.388978 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca\") pod \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " Nov 03 00:44:09.389216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.389067 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca\") pod \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " Nov 03 00:44:09.389216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.389098 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8\" (UniqueName: \"kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8\") pod \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\" (UID: \"5745d317-84cc-4bf2-9a4d-1017245c2cbf\") " Nov 03 00:44:09.389310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.389283 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca" pod "5745d317-84cc-4bf2-9a4d-1017245c2cbf" (UID: "5745d317-84cc-4bf2-9a4d-1017245c2cbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:09.389485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.389464 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:09.391449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.391419 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca" pod "5745d317-84cc-4bf2-9a4d-1017245c2cbf" (UID: "5745d317-84cc-4bf2-9a4d-1017245c2cbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:09.391449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.391431 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8" pod "5745d317-84cc-4bf2-9a4d-1017245c2cbf" (UID: "5745d317-84cc-4bf2-9a4d-1017245c2cbf"). InnerVolumeSpecName "kube-api-access-5vsn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:09.490771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.490740 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5745d317-84cc-4bf2-9a4d-1017245c2cbf-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:09.490771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:09.490767 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vsn8\" (UniqueName: \"kubernetes.io/projected/5745d317-84cc-4bf2-9a4d-1017245c2cbf-kube-api-access-5vsn8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:10.091573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:10.091536 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body= Nov 03 00:44:10.091812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:10.091599 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" Nov 03 00:44:10.096191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:10.096163 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" Nov 03 00:44:10.096326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:10.096208 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4" event={"ID":"5745d317-84cc-4bf2-9a4d-1017245c2cbf","Type":"ContainerDied","Data":"55661872ef8dfa21f48f3b53e0bccd3863759dbe401b73229e76ab83ff838543"} Nov 03 00:44:10.096326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:10.096270 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55661872ef8dfa21f48f3b53e0bccd3863759dbe401b73229e76ab83ff838543" Nov 03 00:44:11.097769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:11.097739 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:11.098395 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:11.098370 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:11.099214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:11.099193 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:44:12.157072 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:44:12.157042 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:44:17.525466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.525433 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8"] Nov 03 00:44:17.526009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.525992 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5745d317-84cc-4bf2-9a4d-1017245c2cbf" containerName="revoke-keys" Nov 03 00:44:17.526058 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.526009 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5745d317-84cc-4bf2-9a4d-1017245c2cbf" containerName="revoke-keys" Nov 03 00:44:17.526113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.526104 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5745d317-84cc-4bf2-9a4d-1017245c2cbf" containerName="revoke-keys" Nov 03 00:44:17.530700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.530685 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.578785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.578761 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sb42z\" (UniqueName: \"kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.578895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.578803 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.578895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.578883 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.584831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.584815 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:17.634490 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.634467 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:44:17.638307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.638283 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8"] Nov 03 00:44:17.680230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.680206 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.680333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.680252 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.680379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.680345 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sb42z\" (UniqueName: \"kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.680847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.680829 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.682835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.682811 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.737224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.737201 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sb42z\" (UniqueName: \"kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z\") pod \"maas-api-revoke-sub-6a3d56fe-zmpj8\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:17.843785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:17.843729 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:18.219211 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:18.219181 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146b5fbd_12ff_49ca_b6d1_81d24e85ec8a.slice/crio-493a71927bed34c4bb759c0e3cb631ee051411aedca577ad59e1a845095a1387 WatchSource:0}: Error finding container 493a71927bed34c4bb759c0e3cb631ee051411aedca577ad59e1a845095a1387: Status 404 returned error can't find the container with id 493a71927bed34c4bb759c0e3cb631ee051411aedca577ad59e1a845095a1387 Nov 03 00:44:18.219477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:18.219449 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8"] Nov 03 00:44:19.124662 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.124250 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d"] Nov 03 00:44:19.132369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.130441 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.166843 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.166810 5144 generic.go:356] "Generic (PLEG): container finished" podID="146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" containerID="450838d0a1b3fed8967bb4ec4b4dbd0ada622b414d2f988e436b886052dd450c" exitCode=0 Nov 03 00:44:19.167001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.166895 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" event={"ID":"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a","Type":"ContainerDied","Data":"450838d0a1b3fed8967bb4ec4b4dbd0ada622b414d2f988e436b886052dd450c"} Nov 03 00:44:19.167001 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.166958 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" event={"ID":"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a","Type":"ContainerStarted","Data":"493a71927bed34c4bb759c0e3cb631ee051411aedca577ad59e1a845095a1387"} Nov 03 00:44:19.185326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.185299 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d"] Nov 03 00:44:19.200712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.200681 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.200859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.200817 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bff8b\" (UniqueName: \"kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.200931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.200918 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.303098 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.303068 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bff8b\" (UniqueName: \"kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.303235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.303125 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.303235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.303189 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.303916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.303837 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.305534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.305515 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.359408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.359353 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bff8b\" (UniqueName: \"kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b\") pod \"maas-api-revoke-sub-43342ee6-d628d\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.460233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.460210 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:19.838234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:19.838207 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d"] Nov 03 00:44:19.838357 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:19.838221 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f8da45_cd7b_4e11_9d35_70559dcae73b.slice/crio-1b0a6ace7d56e244a8ae1db2a587e6e8350804cdf71a3973d8876dcae329458a WatchSource:0}: Error finding container 1b0a6ace7d56e244a8ae1db2a587e6e8350804cdf71a3973d8876dcae329458a: Status 404 returned error can't find the container with id 1b0a6ace7d56e244a8ae1db2a587e6e8350804cdf71a3973d8876dcae329458a Nov 03 00:44:20.174718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.174682 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" event={"ID":"47f8da45-cd7b-4e11-9d35-70559dcae73b","Type":"ContainerStarted","Data":"f75b8bf9bd35169df6061ed2c56fa08a2def105003ab51d7391ca0deb3d621c2"} Nov 03 00:44:20.175046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.174745 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" event={"ID":"47f8da45-cd7b-4e11-9d35-70559dcae73b","Type":"ContainerStarted","Data":"1b0a6ace7d56e244a8ae1db2a587e6e8350804cdf71a3973d8876dcae329458a"} Nov 03 00:44:20.288390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.288342 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" podStartSLOduration=1.288326683 podStartE2EDuration="1.288326683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:20.286892445 +0000 UTC m=+2350.432586901" watchObservedRunningTime="2026-11-03 00:44:20.288326683 +0000 UTC m=+2350.434021139" Nov 03 00:44:20.309766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.309743 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:20.415621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.415555 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z\" (UniqueName: \"kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z\") pod \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " Nov 03 00:44:20.415788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.415707 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca\") pod \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " Nov 03 00:44:20.415855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.415818 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca\") pod \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\" (UID: \"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a\") " Nov 03 00:44:20.416171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.416145 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca" pod "146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" (UID: "146b5fbd-12ff-49ca-b6d1-81d24e85ec8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:20.417758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.417738 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z" pod "146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" (UID: "146b5fbd-12ff-49ca-b6d1-81d24e85ec8a"). InnerVolumeSpecName "kube-api-access-sb42z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:20.418132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.418113 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca" pod "146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" (UID: "146b5fbd-12ff-49ca-b6d1-81d24e85ec8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:20.517203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.517168 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:20.517203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.517195 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:20.517203 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:20.517206 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sb42z\" (UniqueName: \"kubernetes.io/projected/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a-kube-api-access-sb42z\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:21.185394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:21.185361 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" Nov 03 00:44:21.185854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:21.185393 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8" event={"ID":"146b5fbd-12ff-49ca-b6d1-81d24e85ec8a","Type":"ContainerDied","Data":"493a71927bed34c4bb759c0e3cb631ee051411aedca577ad59e1a845095a1387"} Nov 03 00:44:21.185854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:21.185438 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="493a71927bed34c4bb759c0e3cb631ee051411aedca577ad59e1a845095a1387" Nov 03 00:44:21.187268 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:21.187244 5144 generic.go:356] "Generic (PLEG): container finished" podID="47f8da45-cd7b-4e11-9d35-70559dcae73b" containerID="f75b8bf9bd35169df6061ed2c56fa08a2def105003ab51d7391ca0deb3d621c2" exitCode=0 Nov 03 00:44:21.187415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:21.187331 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" event={"ID":"47f8da45-cd7b-4e11-9d35-70559dcae73b","Type":"ContainerDied","Data":"f75b8bf9bd35169df6061ed2c56fa08a2def105003ab51d7391ca0deb3d621c2"} Nov 03 00:44:22.354927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.354902 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:22.440255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.440227 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca\") pod \"47f8da45-cd7b-4e11-9d35-70559dcae73b\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " Nov 03 00:44:22.440403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.440337 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b\" (UniqueName: \"kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b\") pod \"47f8da45-cd7b-4e11-9d35-70559dcae73b\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " Nov 03 00:44:22.440403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.440397 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca\") pod \"47f8da45-cd7b-4e11-9d35-70559dcae73b\" (UID: \"47f8da45-cd7b-4e11-9d35-70559dcae73b\") " Nov 03 00:44:22.440602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.440577 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca" pod "47f8da45-cd7b-4e11-9d35-70559dcae73b" (UID: "47f8da45-cd7b-4e11-9d35-70559dcae73b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:22.440824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.440801 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:22.442598 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.442567 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca" pod "47f8da45-cd7b-4e11-9d35-70559dcae73b" (UID: "47f8da45-cd7b-4e11-9d35-70559dcae73b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:22.442775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.442603 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b" pod "47f8da45-cd7b-4e11-9d35-70559dcae73b" (UID: "47f8da45-cd7b-4e11-9d35-70559dcae73b"). InnerVolumeSpecName "kube-api-access-bff8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:22.542314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.542278 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bff8b\" (UniqueName: \"kubernetes.io/projected/47f8da45-cd7b-4e11-9d35-70559dcae73b-kube-api-access-bff8b\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:22.542314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:22.542308 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/47f8da45-cd7b-4e11-9d35-70559dcae73b-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:23.202642 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:23.202600 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" Nov 03 00:44:23.202819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:23.202623 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d" event={"ID":"47f8da45-cd7b-4e11-9d35-70559dcae73b","Type":"ContainerDied","Data":"1b0a6ace7d56e244a8ae1db2a587e6e8350804cdf71a3973d8876dcae329458a"} Nov 03 00:44:23.202819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:23.202696 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b0a6ace7d56e244a8ae1db2a587e6e8350804cdf71a3973d8876dcae329458a" Nov 03 00:44:29.034081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034051 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46"] Nov 03 00:44:29.034605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034592 5144 cpu_manager.go:414] "containerMap: removing container" podUID="146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" containerName="revoke-keys" Nov 03 00:44:29.034671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034606 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" containerName="revoke-keys" Nov 03 00:44:29.034671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034649 5144 cpu_manager.go:414] "containerMap: removing container" podUID="47f8da45-cd7b-4e11-9d35-70559dcae73b" containerName="revoke-keys" Nov 03 00:44:29.034671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034655 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f8da45-cd7b-4e11-9d35-70559dcae73b" containerName="revoke-keys" Nov 03 00:44:29.034818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034747 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f8da45-cd7b-4e11-9d35-70559dcae73b" containerName="revoke-keys" Nov 03 00:44:29.034818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.034760 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" containerName="revoke-keys" Nov 03 00:44:29.038680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.038622 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.094686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.094663 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46"] Nov 03 00:44:29.112126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.112100 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.112271 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.112174 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfdhr\" (UniqueName: \"kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.112341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.112280 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.139059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.139036 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:29.139177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.139122 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:44:29.213332 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.213304 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.213466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.213398 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfdhr\" (UniqueName: \"kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.213466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.213450 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.214172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.214090 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.215865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.215842 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.269783 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.269756 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfdhr\" (UniqueName: \"kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr\") pod \"maas-api-revoke-sub-e2c5035b-zzg46\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.351943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.351878 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:29.736675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:29.731530 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46"] Nov 03 00:44:29.736675 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:29.732146 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1742329_5a3d_455d_a189_8c1ddcaee582.slice/crio-65e721613d0c29a5e1ad66a19ebefe511fda6537dd73b1b045c9c321cd6ee489 WatchSource:0}: Error finding container 65e721613d0c29a5e1ad66a19ebefe511fda6537dd73b1b045c9c321cd6ee489: Status 404 returned error can't find the container with id 65e721613d0c29a5e1ad66a19ebefe511fda6537dd73b1b045c9c321cd6ee489 Nov 03 00:44:30.268760 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:30.268724 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" event={"ID":"f1742329-5a3d-455d-a189-8c1ddcaee582","Type":"ContainerStarted","Data":"8d7a4220f4b4ae99e7e90ee38f7424f93534e8a7ed31d09da0d69a3c67bd3bc8"} Nov 03 00:44:30.269109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:30.268785 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" event={"ID":"f1742329-5a3d-455d-a189-8c1ddcaee582","Type":"ContainerStarted","Data":"65e721613d0c29a5e1ad66a19ebefe511fda6537dd73b1b045c9c321cd6ee489"} Nov 03 00:44:30.379981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:30.379877 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" podStartSLOduration=2.379858906 podStartE2EDuration="2.379858906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:30.379517519 +0000 UTC m=+2360.525211976" watchObservedRunningTime="2026-11-03 00:44:30.379858906 +0000 UTC m=+2360.525553364" Nov 03 00:44:31.277228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.277196 5144 generic.go:356] "Generic (PLEG): container finished" podID="f1742329-5a3d-455d-a189-8c1ddcaee582" containerID="8d7a4220f4b4ae99e7e90ee38f7424f93534e8a7ed31d09da0d69a3c67bd3bc8" exitCode=0 Nov 03 00:44:31.277659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.277234 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" event={"ID":"f1742329-5a3d-455d-a189-8c1ddcaee582","Type":"ContainerDied","Data":"8d7a4220f4b4ae99e7e90ee38f7424f93534e8a7ed31d09da0d69a3c67bd3bc8"} Nov 03 00:44:31.389373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.389339 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc"] Nov 03 00:44:31.393565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.393546 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.450083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.449394 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc"] Nov 03 00:44:31.557430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.557347 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5msnq\" (UniqueName: \"kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.557430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.557394 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.557653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.557607 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.659002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.658963 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.659212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.659065 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5msnq\" (UniqueName: \"kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.659212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.659096 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.659757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.659729 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.661528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.661505 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:31.716092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:31.716060 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5msnq\" (UniqueName: \"kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq\") pod \"maas-api-revoke-sub-41790f4e-vbmjc\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:32.006482 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.006458 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:32.387659 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.387613 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc"] Nov 03 00:44:32.443102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.443076 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:32.568796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.568715 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca\") pod \"f1742329-5a3d-455d-a189-8c1ddcaee582\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " Nov 03 00:44:32.568796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.568759 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr\" (UniqueName: \"kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr\") pod \"f1742329-5a3d-455d-a189-8c1ddcaee582\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " Nov 03 00:44:32.569029 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.568804 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca\") pod \"f1742329-5a3d-455d-a189-8c1ddcaee582\" (UID: \"f1742329-5a3d-455d-a189-8c1ddcaee582\") " Nov 03 00:44:32.569328 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.569279 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca" pod "f1742329-5a3d-455d-a189-8c1ddcaee582" (UID: "f1742329-5a3d-455d-a189-8c1ddcaee582"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:32.571425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.571398 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca" pod "f1742329-5a3d-455d-a189-8c1ddcaee582" (UID: "f1742329-5a3d-455d-a189-8c1ddcaee582"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:32.571618 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.571600 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr" pod "f1742329-5a3d-455d-a189-8c1ddcaee582" (UID: "f1742329-5a3d-455d-a189-8c1ddcaee582"). InnerVolumeSpecName "kube-api-access-dfdhr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:32.670752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.670663 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:32.670752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.670701 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfdhr\" (UniqueName: \"kubernetes.io/projected/f1742329-5a3d-455d-a189-8c1ddcaee582-kube-api-access-dfdhr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:32.670752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:32.670717 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1742329-5a3d-455d-a189-8c1ddcaee582-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:33.305313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.303383 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" event={"ID":"3832aeb6-32ce-4bc7-8e55-1a4070abf13f","Type":"ContainerStarted","Data":"37f6e877eda9b637049524010527542426757dd9b1f8e98a4b8e0c49da085ba3"} Nov 03 00:44:33.305313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.303451 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" event={"ID":"3832aeb6-32ce-4bc7-8e55-1a4070abf13f","Type":"ContainerStarted","Data":"805652c183cb3acd0e53c02ec53a03ab9e28a4a5715d427e848665e186f4c31d"} Nov 03 00:44:33.308375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.308350 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" Nov 03 00:44:33.308511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.308338 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46" event={"ID":"f1742329-5a3d-455d-a189-8c1ddcaee582","Type":"ContainerDied","Data":"65e721613d0c29a5e1ad66a19ebefe511fda6537dd73b1b045c9c321cd6ee489"} Nov 03 00:44:33.308511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.308482 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65e721613d0c29a5e1ad66a19ebefe511fda6537dd73b1b045c9c321cd6ee489" Nov 03 00:44:33.464330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.464279 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" podStartSLOduration=2.464265834 podStartE2EDuration="2.464265834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:33.464021561 +0000 UTC m=+2363.609716017" watchObservedRunningTime="2026-11-03 00:44:33.464265834 +0000 UTC m=+2363.609960289" Nov 03 00:44:33.842738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.842704 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7"] Nov 03 00:44:33.843567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.843540 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f1742329-5a3d-455d-a189-8c1ddcaee582" containerName="revoke-keys" Nov 03 00:44:33.843687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.843570 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1742329-5a3d-455d-a189-8c1ddcaee582" containerName="revoke-keys" Nov 03 00:44:33.843819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.843805 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1742329-5a3d-455d-a189-8c1ddcaee582" containerName="revoke-keys" Nov 03 00:44:33.847829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.847809 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.899708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.899684 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-b2e795-openshift-default-dockercfg-f8cc5\"" Nov 03 00:44:33.959913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.947968 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7"] Nov 03 00:44:33.989421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.989379 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.989616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.989442 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992192 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gddb\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992276 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992331 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992366 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992411 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992486 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:33.992591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:33.992528 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094249 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gddb\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094317 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094384 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094415 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094457 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094517 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094577 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094551 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094731 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.094910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.094765 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.095528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.095157 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.095528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.095180 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.095528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.095295 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.095528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.095330 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.095528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.095413 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.097130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.097113 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.097497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.097481 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.150983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.150928 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gddb\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.152503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.152464 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token\") pod \"e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.162179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.162153 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:34.317425 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.317389 5144 generic.go:356] "Generic (PLEG): container finished" podID="3832aeb6-32ce-4bc7-8e55-1a4070abf13f" containerID="37f6e877eda9b637049524010527542426757dd9b1f8e98a4b8e0c49da085ba3" exitCode=0 Nov 03 00:44:34.317579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.317474 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" event={"ID":"3832aeb6-32ce-4bc7-8e55-1a4070abf13f","Type":"ContainerDied","Data":"37f6e877eda9b637049524010527542426757dd9b1f8e98a4b8e0c49da085ba3"} Nov 03 00:44:34.594665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.594607 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7"] Nov 03 00:44:34.595073 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:34.595049 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bf746e0_37b0_4325_b5b7_31465218f238.slice/crio-e209917d58f4c47dbad1f2563541315e370bfcae1aad3a45be11731b0ff15fd9 WatchSource:0}: Error finding container e209917d58f4c47dbad1f2563541315e370bfcae1aad3a45be11731b0ff15fd9: Status 404 returned error can't find the container with id e209917d58f4c47dbad1f2563541315e370bfcae1aad3a45be11731b0ff15fd9 Nov 03 00:44:34.597556 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.597523 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:34.597650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.597588 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:34.597650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:34.597614 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:35.327160 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.327125 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" event={"ID":"5bf746e0-37b0-4325-b5b7-31465218f238","Type":"ContainerStarted","Data":"0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22"} Nov 03 00:44:35.327349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.327181 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" event={"ID":"5bf746e0-37b0-4325-b5b7-31465218f238","Type":"ContainerStarted","Data":"e209917d58f4c47dbad1f2563541315e370bfcae1aad3a45be11731b0ff15fd9"} Nov 03 00:44:35.462172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.462106 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podStartSLOduration=2.462087724 podStartE2EDuration="2.462087724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:35.459595144 +0000 UTC m=+2365.605289600" watchObservedRunningTime="2026-11-03 00:44:35.462087724 +0000 UTC m=+2365.607782184" Nov 03 00:44:35.491587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.491564 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:35.611033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.610963 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca\") pod \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " Nov 03 00:44:35.611033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.610999 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq\" (UniqueName: \"kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq\") pod \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " Nov 03 00:44:35.611530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.611055 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca\") pod \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\" (UID: \"3832aeb6-32ce-4bc7-8e55-1a4070abf13f\") " Nov 03 00:44:35.611530 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.611362 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca" pod "3832aeb6-32ce-4bc7-8e55-1a4070abf13f" (UID: "3832aeb6-32ce-4bc7-8e55-1a4070abf13f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:35.611971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.611950 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:35.613475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.613450 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq" pod "3832aeb6-32ce-4bc7-8e55-1a4070abf13f" (UID: "3832aeb6-32ce-4bc7-8e55-1a4070abf13f"). InnerVolumeSpecName "kube-api-access-5msnq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:35.613645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.613603 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca" pod "3832aeb6-32ce-4bc7-8e55-1a4070abf13f" (UID: "3832aeb6-32ce-4bc7-8e55-1a4070abf13f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:35.693699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.693673 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747"] Nov 03 00:44:35.694238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.694225 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3832aeb6-32ce-4bc7-8e55-1a4070abf13f" containerName="revoke-keys" Nov 03 00:44:35.694307 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.694239 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3832aeb6-32ce-4bc7-8e55-1a4070abf13f" containerName="revoke-keys" Nov 03 00:44:35.694376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.694344 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3832aeb6-32ce-4bc7-8e55-1a4070abf13f" containerName="revoke-keys" Nov 03 00:44:35.698315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.698294 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.713521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.713497 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5msnq\" (UniqueName: \"kubernetes.io/projected/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-kube-api-access-5msnq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:35.713607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.713527 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3832aeb6-32ce-4bc7-8e55-1a4070abf13f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:35.754520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.754495 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747"] Nov 03 00:44:35.814917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.814887 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.815076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.814995 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.815076 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.815038 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngtj2\" (UniqueName: \"kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.916020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.915952 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.916020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.916011 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.916181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.916047 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngtj2\" (UniqueName: \"kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.916684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.916616 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.918389 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.918361 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:35.972294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:35.972271 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngtj2\" (UniqueName: \"kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2\") pod \"maas-api-revoke-sub-3d2a92a0-fd747\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:36.012100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.012081 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:36.162475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.162446 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:36.167514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.167460 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:36.337892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.337863 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" Nov 03 00:44:36.338174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.337890 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc" event={"ID":"3832aeb6-32ce-4bc7-8e55-1a4070abf13f","Type":"ContainerDied","Data":"805652c183cb3acd0e53c02ec53a03ab9e28a4a5715d427e848665e186f4c31d"} Nov 03 00:44:36.338265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.338206 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="805652c183cb3acd0e53c02ec53a03ab9e28a4a5715d427e848665e186f4c31d" Nov 03 00:44:36.338892 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.338870 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:36.341690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.341667 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:44:36.399220 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:36.399188 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod143ed4f6_71c3_41e7_9a4d_7e1b7dd17489.slice/crio-8ff98b5e20af73de6c822737382dd6155bb09944ea959bbd63074fd02aee3ff1 WatchSource:0}: Error finding container 8ff98b5e20af73de6c822737382dd6155bb09944ea959bbd63074fd02aee3ff1: Status 404 returned error can't find the container with id 8ff98b5e20af73de6c822737382dd6155bb09944ea959bbd63074fd02aee3ff1 Nov 03 00:44:36.400214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:36.400192 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747"] Nov 03 00:44:37.347783 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:37.347758 5144 generic.go:356] "Generic (PLEG): container finished" podID="143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" containerID="5fb1fc87f8bbf715491c659ac52bc8a551119ff9d6499011705767a4f4a2df55" exitCode=0 Nov 03 00:44:37.348106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:37.347845 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" event={"ID":"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489","Type":"ContainerDied","Data":"5fb1fc87f8bbf715491c659ac52bc8a551119ff9d6499011705767a4f4a2df55"} Nov 03 00:44:37.348106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:37.347912 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" event={"ID":"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489","Type":"ContainerStarted","Data":"8ff98b5e20af73de6c822737382dd6155bb09944ea959bbd63074fd02aee3ff1"} Nov 03 00:44:38.524776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.524742 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:38.546983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.546965 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca\") pod \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " Nov 03 00:44:38.547089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.547003 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca\") pod \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " Nov 03 00:44:38.547133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.547108 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2\" (UniqueName: \"kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2\") pod \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\" (UID: \"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489\") " Nov 03 00:44:38.547350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.547324 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca" pod "143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" (UID: "143ed4f6-71c3-41e7-9a4d-7e1b7dd17489"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:38.547595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.547573 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:38.549236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.549216 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca" pod "143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" (UID: "143ed4f6-71c3-41e7-9a4d-7e1b7dd17489"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:38.549331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.549268 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2" pod "143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" (UID: "143ed4f6-71c3-41e7-9a4d-7e1b7dd17489"). InnerVolumeSpecName "kube-api-access-ngtj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:38.648686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.648597 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:38.648686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:38.648657 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngtj2\" (UniqueName: \"kubernetes.io/projected/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489-kube-api-access-ngtj2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:39.365011 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:39.364980 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" Nov 03 00:44:39.365222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:39.365017 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747" event={"ID":"143ed4f6-71c3-41e7-9a4d-7e1b7dd17489","Type":"ContainerDied","Data":"8ff98b5e20af73de6c822737382dd6155bb09944ea959bbd63074fd02aee3ff1"} Nov 03 00:44:39.365222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:39.365068 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ff98b5e20af73de6c822737382dd6155bb09944ea959bbd63074fd02aee3ff1" Nov 03 00:44:41.882047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.882015 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk"] Nov 03 00:44:41.882819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.882798 5144 cpu_manager.go:414] "containerMap: removing container" podUID="143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" containerName="revoke-keys" Nov 03 00:44:41.882953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.882820 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" containerName="revoke-keys" Nov 03 00:44:41.883019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.883010 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" containerName="revoke-keys" Nov 03 00:44:41.886868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.886848 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:41.943822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.943784 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk"] Nov 03 00:44:41.985714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.985683 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b2e795\"" Nov 03 00:44:41.986014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.985703 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b2e795-dockercfg-gnkgh\"" Nov 03 00:44:41.986491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.986413 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:41.986795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:41.986774 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d78b6\" (UniqueName: \"kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6\") pod \"payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:42.045261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.045232 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp"] Nov 03 00:44:42.062023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.061604 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.078970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.078862 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp"] Nov 03 00:44:42.087643 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.087595 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.087900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.087831 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d78b6\" (UniqueName: \"kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6\") pod \"payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:42.088042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.087924 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb\") pod \"payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.088042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.087968 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:42.088925 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.088879 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:42.144973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.144893 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d78b6\" (UniqueName: \"kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6\") pod \"payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:42.156668 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:44:42.156619 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:44:42.189206 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.189175 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb\") pod \"payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.189336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.189259 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.189960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.189935 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.190890 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.190869 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg"] Nov 03 00:44:42.195213 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.195195 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.200287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.200270 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:42.248293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.248261 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb\") pod \"payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.249815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.249788 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:42.250174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.250156 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:44:42.262917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.262887 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg"] Nov 03 00:44:42.290439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.290412 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzxwf\" (UniqueName: \"kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.290600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.290462 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.290600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.290489 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.391196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.391163 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:42.391742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.391717 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzxwf\" (UniqueName: \"kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.391834 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.391789 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.391834 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.391823 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.392424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.392403 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.394330 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.394309 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.450256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.450227 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzxwf\" (UniqueName: \"kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf\") pod \"maas-api-revoke-sub-9fd792be-7hpfg\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.510204 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.510170 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:42.588902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.588874 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk"] Nov 03 00:44:42.597206 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:42.597181 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee41c01d_7380_4c57_95db_2fed4c7ddbec.slice/crio-39f613fd9bd61aafe5b4e9378b6bdefae9b8c2eba2bc48f15ef6a042401d958f WatchSource:0}: Error finding container 39f613fd9bd61aafe5b4e9378b6bdefae9b8c2eba2bc48f15ef6a042401d958f: Status 404 returned error can't find the container with id 39f613fd9bd61aafe5b4e9378b6bdefae9b8c2eba2bc48f15ef6a042401d958f Nov 03 00:44:42.768072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.768045 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp"] Nov 03 00:44:42.884771 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:42.884741 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55f90be2_5fce_4eaf_8bf2_4faf409c2a33.slice/crio-f1274786b09fe226312b0218cf4f915aa116028008803be11ca55ee5d1ea64d5 WatchSource:0}: Error finding container f1274786b09fe226312b0218cf4f915aa116028008803be11ca55ee5d1ea64d5: Status 404 returned error can't find the container with id f1274786b09fe226312b0218cf4f915aa116028008803be11ca55ee5d1ea64d5 Nov 03 00:44:42.884771 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:42.884760 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg"] Nov 03 00:44:43.404844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.404803 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" event={"ID":"55f90be2-5fce-4eaf-8bf2-4faf409c2a33","Type":"ContainerStarted","Data":"0b907d3dbe6a98c80c8b813840495cf5d1392cc436bd88d6b177760e15a38798"} Nov 03 00:44:43.405244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.404868 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" event={"ID":"55f90be2-5fce-4eaf-8bf2-4faf409c2a33","Type":"ContainerStarted","Data":"f1274786b09fe226312b0218cf4f915aa116028008803be11ca55ee5d1ea64d5"} Nov 03 00:44:43.406564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.406535 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" event={"ID":"2a8e5d31-1e1c-4550-b520-33160ad876db","Type":"ContainerStarted","Data":"b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45"} Nov 03 00:44:43.406726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.406580 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" event={"ID":"2a8e5d31-1e1c-4550-b520-33160ad876db","Type":"ContainerStarted","Data":"2efe41e663a66324407fc41352effa172d3bdacad9c00e083cbcf757c18e8b84"} Nov 03 00:44:43.406726 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.406608 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:43.407999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.407976 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" event={"ID":"ee41c01d-7380-4c57-95db-2fed4c7ddbec","Type":"ContainerStarted","Data":"b58dc8193beaf3222ca8e249e8e65cb04ba60ac1e8c013cc5bde0651822b681c"} Nov 03 00:44:43.408118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.408013 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" event={"ID":"ee41c01d-7380-4c57-95db-2fed4c7ddbec","Type":"ContainerStarted","Data":"39f613fd9bd61aafe5b4e9378b6bdefae9b8c2eba2bc48f15ef6a042401d958f"} Nov 03 00:44:43.408118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.408041 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:43.518897 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.518848 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" podStartSLOduration=1.518834455 podStartE2EDuration="1.518834455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:43.51871468 +0000 UTC m=+2373.664409139" watchObservedRunningTime="2026-11-03 00:44:43.518834455 +0000 UTC m=+2373.664528911" Nov 03 00:44:43.635874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.635827 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" podStartSLOduration=2.635814286 podStartE2EDuration="2.635814286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:43.632744162 +0000 UTC m=+2373.778438636" watchObservedRunningTime="2026-11-03 00:44:43.635814286 +0000 UTC m=+2373.781508762" Nov 03 00:44:43.669465 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.669440 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l"] Nov 03 00:44:43.673878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.673861 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708414 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708446 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708535 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708471 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708671 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708746 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfbmn\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708806 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708833 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708850 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.708945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.708888 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.728324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.728307 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7bfc2-openshift-default-dockercfg-g56sm\"" Nov 03 00:44:43.754919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.754800 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" podStartSLOduration=2.75478188 podStartE2EDuration="2.75478188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:43.749534554 +0000 UTC m=+2373.895229029" watchObservedRunningTime="2026-11-03 00:44:43.75478188 +0000 UTC m=+2373.900476522" Nov 03 00:44:43.763768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.762000 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l"] Nov 03 00:44:43.809736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809708 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.809904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809754 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfbmn\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.809904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809817 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.809904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809847 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.809904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809875 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809912 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.809965 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810016 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810051 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810063 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810336 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810488 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810616 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.810847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.810741 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.813113 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.813092 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.813366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.813342 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.870280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.870212 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.873217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.873185 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfbmn\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:43.988146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:43.988104 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:44.417822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:44.417789 5144 generic.go:356] "Generic (PLEG): container finished" podID="55f90be2-5fce-4eaf-8bf2-4faf409c2a33" containerID="0b907d3dbe6a98c80c8b813840495cf5d1392cc436bd88d6b177760e15a38798" exitCode=0 Nov 03 00:44:44.417980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:44.417886 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" event={"ID":"55f90be2-5fce-4eaf-8bf2-4faf409c2a33","Type":"ContainerDied","Data":"0b907d3dbe6a98c80c8b813840495cf5d1392cc436bd88d6b177760e15a38798"} Nov 03 00:44:44.419015 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:44.418989 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfe14db8_a61e_4aa7_a314_c7e514099414.slice/crio-98502ad68660b07667aeb42e6fae4b1f18a6ffcc7b98dbb1e709f264fa3320df WatchSource:0}: Error finding container 98502ad68660b07667aeb42e6fae4b1f18a6ffcc7b98dbb1e709f264fa3320df: Status 404 returned error can't find the container with id 98502ad68660b07667aeb42e6fae4b1f18a6ffcc7b98dbb1e709f264fa3320df Nov 03 00:44:44.420531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:44.420469 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l"] Nov 03 00:44:44.421375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:44.421337 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:44.421484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:44.421411 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:44.421484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:44.421453 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:44:45.447455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.447419 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" event={"ID":"dfe14db8-a61e-4aa7-a314-c7e514099414","Type":"ContainerStarted","Data":"0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302"} Nov 03 00:44:45.447980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.447475 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" event={"ID":"dfe14db8-a61e-4aa7-a314-c7e514099414","Type":"ContainerStarted","Data":"98502ad68660b07667aeb42e6fae4b1f18a6ffcc7b98dbb1e709f264fa3320df"} Nov 03 00:44:45.571671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.567780 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podStartSLOduration=2.567762155 podStartE2EDuration="2.567762155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:45.563401328 +0000 UTC m=+2375.709095802" watchObservedRunningTime="2026-11-03 00:44:45.567762155 +0000 UTC m=+2375.713456610" Nov 03 00:44:45.651794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.651768 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:45.751446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.751364 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca\") pod \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " Nov 03 00:44:45.751605 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.751579 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca\") pod \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " Nov 03 00:44:45.751692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.751676 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf\" (UniqueName: \"kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf\") pod \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\" (UID: \"55f90be2-5fce-4eaf-8bf2-4faf409c2a33\") " Nov 03 00:44:45.751770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.751750 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca" pod "55f90be2-5fce-4eaf-8bf2-4faf409c2a33" (UID: "55f90be2-5fce-4eaf-8bf2-4faf409c2a33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:44:45.752082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.752061 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:45.753853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.753831 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca" pod "55f90be2-5fce-4eaf-8bf2-4faf409c2a33" (UID: "55f90be2-5fce-4eaf-8bf2-4faf409c2a33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:44:45.753963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.753858 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf" pod "55f90be2-5fce-4eaf-8bf2-4faf409c2a33" (UID: "55f90be2-5fce-4eaf-8bf2-4faf409c2a33"). InnerVolumeSpecName "kube-api-access-xzxwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:44:45.852779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.852737 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:45.852779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.852779 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzxwf\" (UniqueName: \"kubernetes.io/projected/55f90be2-5fce-4eaf-8bf2-4faf409c2a33-kube-api-access-xzxwf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:44:45.990220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.990181 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 00:44:45.990405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.990241 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 00:44:45.997624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:45.997590 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:46.461080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:46.460998 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" Nov 03 00:44:46.461080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:46.461027 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg" event={"ID":"55f90be2-5fce-4eaf-8bf2-4faf409c2a33","Type":"ContainerDied","Data":"f1274786b09fe226312b0218cf4f915aa116028008803be11ca55ee5d1ea64d5"} Nov 03 00:44:46.461080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:46.461081 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1274786b09fe226312b0218cf4f915aa116028008803be11ca55ee5d1ea64d5" Nov 03 00:44:46.988581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:46.988545 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 00:44:46.988833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:46.988596 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 00:44:47.988703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:47.988670 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 00:44:47.989121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:47.988716 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 00:44:48.988728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:48.988624 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 00:44:48.988728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:48.988709 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 00:44:49.989867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:49.989828 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 00:44:49.990380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:49.989913 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 00:44:50.989123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:50.989087 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" start-of-body= Nov 03 00:44:50.989329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:50.989146 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.226:15021/healthz/ready\": dial tcp 10.132.0.226:15021: connect: connection refused" Nov 03 00:44:51.997461 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:51.997431 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:51.997918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:51.997901 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:51.998732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:51.998711 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:44:54.421177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:54.421145 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 00:44:54.421687 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:54.421219 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 00:44:57.975385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:57.975347 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw"] Nov 03 00:44:57.976023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:57.976007 5144 cpu_manager.go:414] "containerMap: removing container" podUID="55f90be2-5fce-4eaf-8bf2-4faf409c2a33" containerName="revoke-keys" Nov 03 00:44:57.976088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:57.976025 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55f90be2-5fce-4eaf-8bf2-4faf409c2a33" containerName="revoke-keys" Nov 03 00:44:57.976129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:57.976112 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="55f90be2-5fce-4eaf-8bf2-4faf409c2a33" containerName="revoke-keys" Nov 03 00:44:57.979844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:57.979821 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.041221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.041173 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw"] Nov 03 00:44:58.083672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.083644 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:44:58.083672 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.083622 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:44:58.143910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.143876 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.144078 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.144053 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn7t7\" (UniqueName: \"kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.144162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.144104 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.244860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.244793 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.244998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.244898 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.244998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.244962 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn7t7\" (UniqueName: \"kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.245616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.245591 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.247417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.247393 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.301878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.301851 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn7t7\" (UniqueName: \"kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7\") pod \"maas-api-revoke-sub-487dea6d-tv6qw\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.593921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.593892 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:44:58.975418 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:44:58.975386 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c396136_969a_4515_bfc1_e2dd538eb5bc.slice/crio-c7fd1e2c6e77416a0e236b4dc8c56b7712e73829ed254eccb2099738c89304f4 WatchSource:0}: Error finding container c7fd1e2c6e77416a0e236b4dc8c56b7712e73829ed254eccb2099738c89304f4: Status 404 returned error can't find the container with id c7fd1e2c6e77416a0e236b4dc8c56b7712e73829ed254eccb2099738c89304f4 Nov 03 00:44:58.975889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:58.975868 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw"] Nov 03 00:44:59.577464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:59.577430 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" event={"ID":"6c396136-969a-4515-bfc1-e2dd538eb5bc","Type":"ContainerStarted","Data":"729778112f0e6b665602313223855030be1af0e53986285c3a5d923fb39b39d3"} Nov 03 00:44:59.577677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:59.577487 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" event={"ID":"6c396136-969a-4515-bfc1-e2dd538eb5bc","Type":"ContainerStarted","Data":"c7fd1e2c6e77416a0e236b4dc8c56b7712e73829ed254eccb2099738c89304f4"} Nov 03 00:44:59.691501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:44:59.691449 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" podStartSLOduration=2.691435088 podStartE2EDuration="2.691435088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:59.689204228 +0000 UTC m=+2389.834898685" watchObservedRunningTime="2026-11-03 00:44:59.691435088 +0000 UTC m=+2389.837129579" Nov 03 00:45:00.163114 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.163083 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx"] Nov 03 00:45:00.167358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.167336 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.168738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.168698 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5"] Nov 03 00:45:00.172850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.172828 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz"] Nov 03 00:45:00.172973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.172900 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.176964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.176944 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.177209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.177192 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d"] Nov 03 00:45:00.181043 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.181026 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.206488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.206470 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz"] Nov 03 00:45:00.210365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.210348 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.270520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270488 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.270703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270529 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z6sl\" (UniqueName: \"kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.270703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270610 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjqrn\" (UniqueName: \"kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn\") pod \"maas-api-key-cleanup-29894445-x9skz\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.270703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270698 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.270885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270726 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.270885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270803 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f9rt\" (UniqueName: \"kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.270885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270845 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-x9skz\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.270988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.270941 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8n85\" (UniqueName: \"kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.299279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.299254 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5"] Nov 03 00:45:00.305448 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.305415 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx"] Nov 03 00:45:00.333994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.333967 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d"] Nov 03 00:45:00.362694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.362660 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz"] Nov 03 00:45:00.372721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.372686 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.372721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.372748 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z6sl\" (UniqueName: \"kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.372791 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjqrn\" (UniqueName: \"kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn\") pod \"maas-api-key-cleanup-29894445-x9skz\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.372824 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.372858 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373039 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f9rt\" (UniqueName: \"kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373089 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-x9skz\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373152 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8n85\" (UniqueName: \"kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373202 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373233 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmz7r\" (UniqueName: \"kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r\") pod \"maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373534 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373624 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.373956 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.374341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.374280 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894445-x9skz\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.384907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.384884 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz"] Nov 03 00:45:00.432574 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.432505 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8n85\" (UniqueName: \"kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.434300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.434273 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f9rt\" (UniqueName: \"kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.434441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.434406 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjqrn\" (UniqueName: \"kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn\") pod \"maas-api-key-cleanup-29894445-x9skz\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.434880 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.434857 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z6sl\" (UniqueName: \"kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.474321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.474297 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.474449 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.474328 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmz7r\" (UniqueName: \"kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r\") pod \"maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.474886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.474867 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.483015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.482998 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:00.490046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.490030 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:00.497035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.497011 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:00.502871 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.502852 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:00.537652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.533281 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmz7r\" (UniqueName: \"kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r\") pod \"maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.607414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.607272 5144 generic.go:356] "Generic (PLEG): container finished" podID="6c396136-969a-4515-bfc1-e2dd538eb5bc" containerID="729778112f0e6b665602313223855030be1af0e53986285c3a5d923fb39b39d3" exitCode=0 Nov 03 00:45:00.607414 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.607347 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" event={"ID":"6c396136-969a-4515-bfc1-e2dd538eb5bc","Type":"ContainerDied","Data":"729778112f0e6b665602313223855030be1af0e53986285c3a5d923fb39b39d3"} Nov 03 00:45:00.824833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.824803 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:00.888002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.887969 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx"] Nov 03 00:45:00.942752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.942719 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5"] Nov 03 00:45:00.949600 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:00.949305 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b8fa8b3_4ee9_4a97_8ae2_f49c77a3b9a3.slice/crio-5bd541aeb806d6f2fe88669f4015585d6300f28c846291148e47df7d031684fd WatchSource:0}: Error finding container 5bd541aeb806d6f2fe88669f4015585d6300f28c846291148e47df7d031684fd: Status 404 returned error can't find the container with id 5bd541aeb806d6f2fe88669f4015585d6300f28c846291148e47df7d031684fd Nov 03 00:45:00.958580 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.958554 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d"] Nov 03 00:45:00.966283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:00.966256 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz"] Nov 03 00:45:01.205011 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:01.204985 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64983805_ee5f_423d_9e9c_76a6f0dbea46.slice/crio-18b72985e0e34dd8942d417118ca550fc04f99dafa73e04cd18db399d33c7a19 WatchSource:0}: Error finding container 18b72985e0e34dd8942d417118ca550fc04f99dafa73e04cd18db399d33c7a19: Status 404 returned error can't find the container with id 18b72985e0e34dd8942d417118ca550fc04f99dafa73e04cd18db399d33c7a19 Nov 03 00:45:01.205791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.205749 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz"] Nov 03 00:45:01.618919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.618873 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerStarted","Data":"83c67986ef062925c95356db48dc0a88ebbd8704c6763ddd95b1f39216d20212"} Nov 03 00:45:01.619118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.618951 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerStarted","Data":"5bd541aeb806d6f2fe88669f4015585d6300f28c846291148e47df7d031684fd"} Nov 03 00:45:01.622477 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.622445 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerStarted","Data":"1d5cba6c7781ca27f0647d7cba8b5269bcf25d235b32fbc4d53722afbe51fd81"} Nov 03 00:45:01.622701 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.622503 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerStarted","Data":"429e556d0a143ab4df73b03088d9ec867b829b06a58e5c29b7ff3d08e4722812"} Nov 03 00:45:01.625747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.625711 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerStarted","Data":"e0b56e4536b9ef7db539d60cd0f5ea4de941f575e690da0958265a8d1c754c97"} Nov 03 00:45:01.625951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.625756 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerStarted","Data":"688025752ed9c51b84f92f3dd6f23d3f504be0a6b39b657014d8bc14361369b0"} Nov 03 00:45:01.628840 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.628813 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerStarted","Data":"ede10bfd6f80e6388ae27466c56ab9c78622f247eaa7908897cbe780aa200323"} Nov 03 00:45:01.628956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.628864 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerStarted","Data":"cda7a8cc80c24d548cc47bd03f18dbd1d94a3233559113247f4c166859c62df3"} Nov 03 00:45:01.630502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.630477 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerStarted","Data":"18b72985e0e34dd8942d417118ca550fc04f99dafa73e04cd18db399d33c7a19"} Nov 03 00:45:01.739445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.739381 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" podStartSLOduration=1.739360593 podStartE2EDuration="1.739360593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.73658187 +0000 UTC m=+2391.882276317" watchObservedRunningTime="2026-11-03 00:45:01.739360593 +0000 UTC m=+2391.885055050" Nov 03 00:45:01.796155 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.796128 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:45:01.859761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.859711 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" podStartSLOduration=1.859692868 podStartE2EDuration="1.859692868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.85455851 +0000 UTC m=+2392.000252979" watchObservedRunningTime="2026-11-03 00:45:01.859692868 +0000 UTC m=+2392.005387324" Nov 03 00:45:01.892547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.892514 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca\") pod \"6c396136-969a-4515-bfc1-e2dd538eb5bc\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " Nov 03 00:45:01.892740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.892722 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca\") pod \"6c396136-969a-4515-bfc1-e2dd538eb5bc\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " Nov 03 00:45:01.892816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.892788 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7\" (UniqueName: \"kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7\") pod \"6c396136-969a-4515-bfc1-e2dd538eb5bc\" (UID: \"6c396136-969a-4515-bfc1-e2dd538eb5bc\") " Nov 03 00:45:01.892939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.892912 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca" pod "6c396136-969a-4515-bfc1-e2dd538eb5bc" (UID: "6c396136-969a-4515-bfc1-e2dd538eb5bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:01.893320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.893252 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:01.894884 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.894858 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca" pod "6c396136-969a-4515-bfc1-e2dd538eb5bc" (UID: "6c396136-969a-4515-bfc1-e2dd538eb5bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:01.894965 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.894907 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7" pod "6c396136-969a-4515-bfc1-e2dd538eb5bc" (UID: "6c396136-969a-4515-bfc1-e2dd538eb5bc"). InnerVolumeSpecName "kube-api-access-hn7t7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:01.979931 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.979839 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" podStartSLOduration=1.979825747 podStartE2EDuration="1.979825747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:01.97937614 +0000 UTC m=+2392.125070613" watchObservedRunningTime="2026-11-03 00:45:01.979825747 +0000 UTC m=+2392.125520238" Nov 03 00:45:01.994865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.994841 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c396136-969a-4515-bfc1-e2dd538eb5bc-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:01.994979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:01.994869 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn7t7\" (UniqueName: \"kubernetes.io/projected/6c396136-969a-4515-bfc1-e2dd538eb5bc-kube-api-access-hn7t7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:02.094696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.094626 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" podStartSLOduration=2.094612317 podStartE2EDuration="2.094612317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:02.091041584 +0000 UTC m=+2392.236736040" watchObservedRunningTime="2026-11-03 00:45:02.094612317 +0000 UTC m=+2392.240306764" Nov 03 00:45:02.640264 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.640234 5144 generic.go:356] "Generic (PLEG): container finished" podID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerID="1d5cba6c7781ca27f0647d7cba8b5269bcf25d235b32fbc4d53722afbe51fd81" exitCode=60 Nov 03 00:45:02.640700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.640313 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerDied","Data":"1d5cba6c7781ca27f0647d7cba8b5269bcf25d235b32fbc4d53722afbe51fd81"} Nov 03 00:45:02.640700 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.640532 5144 scope.go:122] "RemoveContainer" containerID="1d5cba6c7781ca27f0647d7cba8b5269bcf25d235b32fbc4d53722afbe51fd81" Nov 03 00:45:02.642740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.642713 5144 generic.go:356] "Generic (PLEG): container finished" podID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerID="e0b56e4536b9ef7db539d60cd0f5ea4de941f575e690da0958265a8d1c754c97" exitCode=60 Nov 03 00:45:02.642855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.642810 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerDied","Data":"e0b56e4536b9ef7db539d60cd0f5ea4de941f575e690da0958265a8d1c754c97"} Nov 03 00:45:02.643010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.642994 5144 scope.go:122] "RemoveContainer" containerID="e0b56e4536b9ef7db539d60cd0f5ea4de941f575e690da0958265a8d1c754c97" Nov 03 00:45:02.654309 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.654279 5144 generic.go:356] "Generic (PLEG): container finished" podID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerID="ede10bfd6f80e6388ae27466c56ab9c78622f247eaa7908897cbe780aa200323" exitCode=60 Nov 03 00:45:02.654459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.654438 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerDied","Data":"ede10bfd6f80e6388ae27466c56ab9c78622f247eaa7908897cbe780aa200323"} Nov 03 00:45:02.654799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.654781 5144 scope.go:122] "RemoveContainer" containerID="ede10bfd6f80e6388ae27466c56ab9c78622f247eaa7908897cbe780aa200323" Nov 03 00:45:02.657694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.657672 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" Nov 03 00:45:02.657838 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.657807 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw" event={"ID":"6c396136-969a-4515-bfc1-e2dd538eb5bc","Type":"ContainerDied","Data":"c7fd1e2c6e77416a0e236b4dc8c56b7712e73829ed254eccb2099738c89304f4"} Nov 03 00:45:02.657963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.657857 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7fd1e2c6e77416a0e236b4dc8c56b7712e73829ed254eccb2099738c89304f4" Nov 03 00:45:02.659793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.659769 5144 generic.go:356] "Generic (PLEG): container finished" podID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerID="0385828785ab0baa925ec8b5a985ddd205d1b58237a9e38617c9c9ccce7524fb" exitCode=60 Nov 03 00:45:02.659979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.659959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerDied","Data":"0385828785ab0baa925ec8b5a985ddd205d1b58237a9e38617c9c9ccce7524fb"} Nov 03 00:45:02.660106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.660094 5144 scope.go:122] "RemoveContainer" containerID="0385828785ab0baa925ec8b5a985ddd205d1b58237a9e38617c9c9ccce7524fb" Nov 03 00:45:02.661712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.661689 5144 generic.go:356] "Generic (PLEG): container finished" podID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerID="83c67986ef062925c95356db48dc0a88ebbd8704c6763ddd95b1f39216d20212" exitCode=60 Nov 03 00:45:02.661809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.661731 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerDied","Data":"83c67986ef062925c95356db48dc0a88ebbd8704c6763ddd95b1f39216d20212"} Nov 03 00:45:02.661917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:02.661904 5144 scope.go:122] "RemoveContainer" containerID="83c67986ef062925c95356db48dc0a88ebbd8704c6763ddd95b1f39216d20212" Nov 03 00:45:03.671071 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.671042 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerStarted","Data":"a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617"} Nov 03 00:45:03.673234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.673207 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerStarted","Data":"8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9"} Nov 03 00:45:03.675170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.675146 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerStarted","Data":"b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280"} Nov 03 00:45:03.677096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.677072 5144 generic.go:356] "Generic (PLEG): container finished" podID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerID="592490c7b3e8408f412ff05b7bd5f97a47769ffb1db7685e0834469cec7cb771" exitCode=60 Nov 03 00:45:03.677212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.677124 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerDied","Data":"592490c7b3e8408f412ff05b7bd5f97a47769ffb1db7685e0834469cec7cb771"} Nov 03 00:45:03.677212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.677163 5144 scope.go:122] "RemoveContainer" containerID="83c67986ef062925c95356db48dc0a88ebbd8704c6763ddd95b1f39216d20212" Nov 03 00:45:03.677453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.677438 5144 scope.go:122] "RemoveContainer" containerID="592490c7b3e8408f412ff05b7bd5f97a47769ffb1db7685e0834469cec7cb771" Nov 03 00:45:03.677739 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:03.677714 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5_odh-ai-gateway-infra(9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" Nov 03 00:45:03.679195 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.679165 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerStarted","Data":"5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9"} Nov 03 00:45:03.837074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.837048 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26"] Nov 03 00:45:03.837688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.837673 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6c396136-969a-4515-bfc1-e2dd538eb5bc" containerName="revoke-keys" Nov 03 00:45:03.837765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.837690 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c396136-969a-4515-bfc1-e2dd538eb5bc" containerName="revoke-keys" Nov 03 00:45:03.837821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.837812 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c396136-969a-4515-bfc1-e2dd538eb5bc" containerName="revoke-keys" Nov 03 00:45:03.841366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.841348 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:03.894822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.894784 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-e4c8b1-dockercfg-fq7mr\"" Nov 03 00:45:03.894822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.894784 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-e4c8b1\"" Nov 03 00:45:03.898098 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.898074 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26"] Nov 03 00:45:03.919528 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:03.919504 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd"] Nov 03 00:45:04.022384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.022324 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" podStartSLOduration=4.022306479 podStartE2EDuration="4.022306479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:04.022226552 +0000 UTC m=+2394.167921008" watchObservedRunningTime="2026-11-03 00:45:04.022306479 +0000 UTC m=+2394.168000937" Nov 03 00:45:04.036054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.036014 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn5dq\" (UniqueName: \"kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq\") pod \"payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.036579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.036557 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.142643 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.142423 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn5dq\" (UniqueName: \"kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq\") pod \"payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.142852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.142780 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.143733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.143705 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.200359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.200324 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn5dq\" (UniqueName: \"kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq\") pod \"payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.272758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.272730 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:04.672821 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:04.672789 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36fc44aa_9b16_4134_b5ce_43105372c5e1.slice/crio-6222045f21e4b41d28763395527414a12a3e7bbc62368d9f5a721c667e1f0302 WatchSource:0}: Error finding container 6222045f21e4b41d28763395527414a12a3e7bbc62368d9f5a721c667e1f0302: Status 404 returned error can't find the container with id 6222045f21e4b41d28763395527414a12a3e7bbc62368d9f5a721c667e1f0302 Nov 03 00:45:04.673415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.673389 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26"] Nov 03 00:45:04.694730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.694697 5144 generic.go:356] "Generic (PLEG): container finished" podID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerID="a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617" exitCode=60 Nov 03 00:45:04.694860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.694757 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerDied","Data":"a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617"} Nov 03 00:45:04.694860 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.694819 5144 scope.go:122] "RemoveContainer" containerID="e0b56e4536b9ef7db539d60cd0f5ea4de941f575e690da0958265a8d1c754c97" Nov 03 00:45:04.696275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.696259 5144 scope.go:122] "RemoveContainer" containerID="a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617" Nov 03 00:45:04.697663 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:04.697616 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx_odh-ai-gateway-infra(4cd7204d-42da-41a9-8dc9-8b7892e05c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" Nov 03 00:45:04.698540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.698520 5144 generic.go:356] "Generic (PLEG): container finished" podID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerID="8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9" exitCode=60 Nov 03 00:45:04.698982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.698966 5144 scope.go:122] "RemoveContainer" containerID="8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9" Nov 03 00:45:04.699248 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:04.699227 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d_odh-ai-gateway-infra(4db2ef9f-1750-41ba-961c-cd866b63da96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" Nov 03 00:45:04.699312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.699274 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerDied","Data":"8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9"} Nov 03 00:45:04.701329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.701306 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" event={"ID":"36fc44aa-9b16-4134-b5ce-43105372c5e1","Type":"ContainerStarted","Data":"6222045f21e4b41d28763395527414a12a3e7bbc62368d9f5a721c667e1f0302"} Nov 03 00:45:04.702878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.702856 5144 generic.go:356] "Generic (PLEG): container finished" podID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerID="b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280" exitCode=60 Nov 03 00:45:04.703012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.702996 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerDied","Data":"b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280"} Nov 03 00:45:04.703272 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.703255 5144 scope.go:122] "RemoveContainer" containerID="b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280" Nov 03 00:45:04.703486 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:04.703466 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz_odh-ai-gateway-infra(64983805-ee5f-423d-9e9c-76a6f0dbea46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" Nov 03 00:45:04.705614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.705596 5144 scope.go:122] "RemoveContainer" containerID="592490c7b3e8408f412ff05b7bd5f97a47769ffb1db7685e0834469cec7cb771" Nov 03 00:45:04.705854 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:04.705830 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5_odh-ai-gateway-infra(9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" Nov 03 00:45:04.707413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.707389 5144 generic.go:356] "Generic (PLEG): container finished" podID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerID="5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9" exitCode=60 Nov 03 00:45:04.707512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.707458 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerDied","Data":"5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9"} Nov 03 00:45:04.707712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.707618 5144 scope.go:122] "RemoveContainer" containerID="5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9" Nov 03 00:45:04.707841 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:04.707824 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-x9skz_odh-ai-gateway-infra(c6b0e6fe-9364-465c-ad69-06216b67d22f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" Nov 03 00:45:04.734769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.734750 5144 scope.go:122] "RemoveContainer" containerID="ede10bfd6f80e6388ae27466c56ab9c78622f247eaa7908897cbe780aa200323" Nov 03 00:45:04.758785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.758762 5144 scope.go:122] "RemoveContainer" containerID="0385828785ab0baa925ec8b5a985ddd205d1b58237a9e38617c9c9ccce7524fb" Nov 03 00:45:04.773964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:04.773940 5144 scope.go:122] "RemoveContainer" containerID="1d5cba6c7781ca27f0647d7cba8b5269bcf25d235b32fbc4d53722afbe51fd81" Nov 03 00:45:05.717590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.717556 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" event={"ID":"36fc44aa-9b16-4134-b5ce-43105372c5e1","Type":"ContainerStarted","Data":"d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547"} Nov 03 00:45:05.718064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.717682 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:05.719887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.719864 5144 scope.go:122] "RemoveContainer" containerID="b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280" Nov 03 00:45:05.720176 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:05.720143 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz_odh-ai-gateway-infra(64983805-ee5f-423d-9e9c-76a6f0dbea46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" Nov 03 00:45:05.721838 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.721815 5144 scope.go:122] "RemoveContainer" containerID="5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9" Nov 03 00:45:05.722067 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:05.722043 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-x9skz_odh-ai-gateway-infra(c6b0e6fe-9364-465c-ad69-06216b67d22f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" Nov 03 00:45:05.723476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.723456 5144 scope.go:122] "RemoveContainer" containerID="a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617" Nov 03 00:45:05.723736 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:05.723712 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx_odh-ai-gateway-infra(4cd7204d-42da-41a9-8dc9-8b7892e05c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" Nov 03 00:45:05.725162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.725147 5144 scope.go:122] "RemoveContainer" containerID="8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9" Nov 03 00:45:05.725336 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:05.725319 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d_odh-ai-gateway-infra(4db2ef9f-1750-41ba-961c-cd866b63da96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" Nov 03 00:45:05.833053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:05.833008 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" podStartSLOduration=2.832995539 podStartE2EDuration="2.832995539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:05.830924685 +0000 UTC m=+2395.976619157" watchObservedRunningTime="2026-11-03 00:45:05.832995539 +0000 UTC m=+2395.978689994" Nov 03 00:45:10.565769 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:10.565737 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:45:10.571193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:10.571167 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:45:12.155760 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:12.155723 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:45:13.988618 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:13.988587 5144 scope.go:122] "RemoveContainer" containerID="8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9" Nov 03 00:45:14.809559 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:14.809521 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerStarted","Data":"c52906a4f9e65f7ff120fc8a4a5343057a3fa81fd90a4daf9965a3e2504c2e3f"} Nov 03 00:45:14.988949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:14.988847 5144 scope.go:122] "RemoveContainer" containerID="592490c7b3e8408f412ff05b7bd5f97a47769ffb1db7685e0834469cec7cb771" Nov 03 00:45:14.989484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:14.989197 5144 scope.go:122] "RemoveContainer" containerID="a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617" Nov 03 00:45:14.989484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:14.989322 5144 scope.go:122] "RemoveContainer" containerID="5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9" Nov 03 00:45:15.820062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.820011 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerStarted","Data":"49068e885e7d4b07835341da3011c0c1bedb35d38f7fd4beaa508325b3cb4380"} Nov 03 00:45:15.822569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.822544 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerStarted","Data":"4f2b6946067a0e7221a7d15454e6cd33b88f80ca19a529f08a3282396f4a1cb1"} Nov 03 00:45:15.824848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.824818 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerStarted","Data":"c1c093698bc87c7cb0138c19a676f4cfba389b13f696cb389ec1cb8e740bfb34"} Nov 03 00:45:15.826957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.826930 5144 generic.go:356] "Generic (PLEG): container finished" podID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerID="c52906a4f9e65f7ff120fc8a4a5343057a3fa81fd90a4daf9965a3e2504c2e3f" exitCode=60 Nov 03 00:45:15.827063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.826995 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerDied","Data":"c52906a4f9e65f7ff120fc8a4a5343057a3fa81fd90a4daf9965a3e2504c2e3f"} Nov 03 00:45:15.827063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.827050 5144 scope.go:122] "RemoveContainer" containerID="8289579f491a75c5a9df95fc5b971c9849303ac88204083382a37252ab63f2a9" Nov 03 00:45:15.827375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.827359 5144 scope.go:122] "RemoveContainer" containerID="c52906a4f9e65f7ff120fc8a4a5343057a3fa81fd90a4daf9965a3e2504c2e3f" Nov 03 00:45:15.827707 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:15.827685 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d_odh-ai-gateway-infra(4db2ef9f-1750-41ba-961c-cd866b63da96)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" Nov 03 00:45:15.956489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:15.956454 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d"] Nov 03 00:45:16.738145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.738114 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 00:45:16.839854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.839818 5144 generic.go:356] "Generic (PLEG): container finished" podID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerID="4f2b6946067a0e7221a7d15454e6cd33b88f80ca19a529f08a3282396f4a1cb1" exitCode=60 Nov 03 00:45:16.840026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.839854 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerDied","Data":"4f2b6946067a0e7221a7d15454e6cd33b88f80ca19a529f08a3282396f4a1cb1"} Nov 03 00:45:16.840026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.839935 5144 scope.go:122] "RemoveContainer" containerID="5aec7c82582b10f0a1c07ac1f0ae191b4be996bd36bc5fdee74a1ae77f63edc9" Nov 03 00:45:16.840324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.840305 5144 scope.go:122] "RemoveContainer" containerID="4f2b6946067a0e7221a7d15454e6cd33b88f80ca19a529f08a3282396f4a1cb1" Nov 03 00:45:16.840699 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:16.840674 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894445-x9skz_odh-ai-gateway-infra(c6b0e6fe-9364-465c-ad69-06216b67d22f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" Nov 03 00:45:16.842517 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.842498 5144 generic.go:356] "Generic (PLEG): container finished" podID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerID="c1c093698bc87c7cb0138c19a676f4cfba389b13f696cb389ec1cb8e740bfb34" exitCode=60 Nov 03 00:45:16.842593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.842564 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerDied","Data":"c1c093698bc87c7cb0138c19a676f4cfba389b13f696cb389ec1cb8e740bfb34"} Nov 03 00:45:16.843021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.843000 5144 scope.go:122] "RemoveContainer" containerID="c1c093698bc87c7cb0138c19a676f4cfba389b13f696cb389ec1cb8e740bfb34" Nov 03 00:45:16.843335 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:16.843309 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx_odh-ai-gateway-infra(4cd7204d-42da-41a9-8dc9-8b7892e05c69)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" Nov 03 00:45:16.847089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.847069 5144 generic.go:356] "Generic (PLEG): container finished" podID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerID="49068e885e7d4b07835341da3011c0c1bedb35d38f7fd4beaa508325b3cb4380" exitCode=60 Nov 03 00:45:16.847201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.847141 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerDied","Data":"49068e885e7d4b07835341da3011c0c1bedb35d38f7fd4beaa508325b3cb4380"} Nov 03 00:45:16.847555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.847538 5144 scope.go:122] "RemoveContainer" containerID="49068e885e7d4b07835341da3011c0c1bedb35d38f7fd4beaa508325b3cb4380" Nov 03 00:45:16.847906 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:16.847884 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5_odh-ai-gateway-infra(9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" Nov 03 00:45:16.859186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.859158 5144 scope.go:122] "RemoveContainer" containerID="a99a769f292437eb58a19a4f8283717e483a7f2153d49f8520e3e198ed3cd617" Nov 03 00:45:16.889284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.889240 5144 scope.go:122] "RemoveContainer" containerID="592490c7b3e8408f412ff05b7bd5f97a47769ffb1db7685e0834469cec7cb771" Nov 03 00:45:16.969886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.953614 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5"] Nov 03 00:45:16.989008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:16.988925 5144 scope.go:122] "RemoveContainer" containerID="b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280" Nov 03 00:45:17.062903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.062874 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:17.068352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.068324 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx"] Nov 03 00:45:17.240049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.239964 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl\" (UniqueName: \"kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl\") pod \"4db2ef9f-1750-41ba-961c-cd866b63da96\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " Nov 03 00:45:17.240259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.240239 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca\") pod \"4db2ef9f-1750-41ba-961c-cd866b63da96\" (UID: \"4db2ef9f-1750-41ba-961c-cd866b63da96\") " Nov 03 00:45:17.240777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.240706 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca" pod "4db2ef9f-1750-41ba-961c-cd866b63da96" (UID: "4db2ef9f-1750-41ba-961c-cd866b63da96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:17.242273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.242249 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl" pod "4db2ef9f-1750-41ba-961c-cd866b63da96" (UID: "4db2ef9f-1750-41ba-961c-cd866b63da96"). InnerVolumeSpecName "kube-api-access-2z6sl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:17.295463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.295431 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz"] Nov 03 00:45:17.342025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.341986 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z6sl\" (UniqueName: \"kubernetes.io/projected/4db2ef9f-1750-41ba-961c-cd866b63da96-kube-api-access-2z6sl\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:17.342025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.342026 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db2ef9f-1750-41ba-961c-cd866b63da96-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:17.866972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.862061 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" Nov 03 00:45:17.866972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.862456 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d" event={"ID":"4db2ef9f-1750-41ba-961c-cd866b63da96","Type":"ContainerDied","Data":"cda7a8cc80c24d548cc47bd03f18dbd1d94a3233559113247f4c166859c62df3"} Nov 03 00:45:17.866972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.862557 5144 scope.go:122] "RemoveContainer" containerID="c52906a4f9e65f7ff120fc8a4a5343057a3fa81fd90a4daf9965a3e2504c2e3f" Nov 03 00:45:17.866972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.864681 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerStarted","Data":"afb83af1940a2c0d3d87d2a832b75cdeacb90d2e3d9695266611fb6ea5ef934e"} Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.915356 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64"] Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.916031 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.916045 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.916091 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.916097 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.916203 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:17.916680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.916212 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:17.921883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.921856 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:17.975368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.975000 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-fce94b\"" Nov 03 00:45:17.977800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.975923 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-fce94b-dockercfg-l8f49\"" Nov 03 00:45:17.982239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:17.982157 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64"] Nov 03 00:45:18.103286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.096302 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.103286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.096522 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p8p9\" (UniqueName: \"kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9\") pod \"payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.103286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.101980 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp"] Nov 03 00:45:18.103286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.102989 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:18.103286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.103006 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:18.103286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.103169 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" containerName="cleanup" Nov 03 00:45:18.113981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.113899 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.134823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.121049 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:18.164124 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.164077 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp"] Nov 03 00:45:18.198142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.197924 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.198245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.198211 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p8p9\" (UniqueName: \"kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9\") pod \"payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.199452 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.199429 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.214590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.214558 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:18.219342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.219310 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:18.258546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.258513 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p8p9\" (UniqueName: \"kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9\") pod \"payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.268615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.268588 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.299388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.299361 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca\") pod \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " Nov 03 00:45:18.299623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.299604 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85\" (UniqueName: \"kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85\") pod \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\" (UID: \"4cd7204d-42da-41a9-8dc9-8b7892e05c69\") " Nov 03 00:45:18.299773 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.299748 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca" pod "4cd7204d-42da-41a9-8dc9-8b7892e05c69" (UID: "4cd7204d-42da-41a9-8dc9-8b7892e05c69"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:18.299971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.299951 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kqb2\" (UniqueName: \"kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2\") pod \"payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.300411 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.299984 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.300411 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.300298 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd7204d-42da-41a9-8dc9-8b7892e05c69-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.303584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.303554 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85" pod "4cd7204d-42da-41a9-8dc9-8b7892e05c69" (UID: "4cd7204d-42da-41a9-8dc9-8b7892e05c69"). InnerVolumeSpecName "kube-api-access-b8n85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:18.400782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.400641 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn\" (UniqueName: \"kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn\") pod \"c6b0e6fe-9364-465c-ad69-06216b67d22f\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " Nov 03 00:45:18.401193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401152 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca\") pod \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " Nov 03 00:45:18.401320 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401274 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt\" (UniqueName: \"kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt\") pod \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\" (UID: \"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3\") " Nov 03 00:45:18.401383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401319 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca\") pod \"c6b0e6fe-9364-465c-ad69-06216b67d22f\" (UID: \"c6b0e6fe-9364-465c-ad69-06216b67d22f\") " Nov 03 00:45:18.401973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401575 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kqb2\" (UniqueName: \"kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2\") pod \"payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.401973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401616 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.401973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401836 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca" pod "9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" (UID: "9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:18.401973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.401927 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca" pod "c6b0e6fe-9364-465c-ad69-06216b67d22f" (UID: "c6b0e6fe-9364-465c-ad69-06216b67d22f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:18.402259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.402227 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.402259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.402256 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6b0e6fe-9364-465c-ad69-06216b67d22f-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.402391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.402272 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8n85\" (UniqueName: \"kubernetes.io/projected/4cd7204d-42da-41a9-8dc9-8b7892e05c69-kube-api-access-b8n85\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.402837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.402769 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.403301 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.403257 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn" pod "c6b0e6fe-9364-465c-ad69-06216b67d22f" (UID: "c6b0e6fe-9364-465c-ad69-06216b67d22f"). InnerVolumeSpecName "kube-api-access-vjqrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:18.404419 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.404392 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt" pod "9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" (UID: "9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3"). InnerVolumeSpecName "kube-api-access-2f9rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:18.441174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.441138 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d"] Nov 03 00:45:18.448053 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.448024 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894445-qd82d"] Nov 03 00:45:18.457706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.457675 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kqb2\" (UniqueName: \"kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2\") pod \"payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.503333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.503304 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2f9rt\" (UniqueName: \"kubernetes.io/projected/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3-kube-api-access-2f9rt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.503456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.503332 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjqrn\" (UniqueName: \"kubernetes.io/projected/c6b0e6fe-9364-465c-ad69-06216b67d22f-kube-api-access-vjqrn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:18.513407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.513374 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:18.664440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.664357 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64"] Nov 03 00:45:18.665035 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:18.665002 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d2f3e00_3bd4_4ab8_b7cc_79daded8a8ac.slice/crio-4578bb6b41a86f4fdb5bbc873a54bd69ae572ac77a407eb018e50f1364c0f3f9 WatchSource:0}: Error finding container 4578bb6b41a86f4fdb5bbc873a54bd69ae572ac77a407eb018e50f1364c0f3f9: Status 404 returned error can't find the container with id 4578bb6b41a86f4fdb5bbc873a54bd69ae572ac77a407eb018e50f1364c0f3f9 Nov 03 00:45:18.877806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.877775 5144 generic.go:356] "Generic (PLEG): container finished" podID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerID="afb83af1940a2c0d3d87d2a832b75cdeacb90d2e3d9695266611fb6ea5ef934e" exitCode=60 Nov 03 00:45:18.878230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.877909 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerDied","Data":"afb83af1940a2c0d3d87d2a832b75cdeacb90d2e3d9695266611fb6ea5ef934e"} Nov 03 00:45:18.878230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.877971 5144 scope.go:122] "RemoveContainer" containerID="b069276ce8dc5b7720ed29e83c8cf76919bd7b3e6499393c0d4e3ca640d57280" Nov 03 00:45:18.878397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.878367 5144 scope.go:122] "RemoveContainer" containerID="afb83af1940a2c0d3d87d2a832b75cdeacb90d2e3d9695266611fb6ea5ef934e" Nov 03 00:45:18.878775 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:18.878754 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz_odh-ai-gateway-infra(64983805-ee5f-423d-9e9c-76a6f0dbea46)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" Nov 03 00:45:18.879909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.879891 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" Nov 03 00:45:18.880004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.879941 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5" event={"ID":"9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3","Type":"ContainerDied","Data":"5bd541aeb806d6f2fe88669f4015585d6300f28c846291148e47df7d031684fd"} Nov 03 00:45:18.881801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.881771 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" event={"ID":"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac","Type":"ContainerStarted","Data":"836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b"} Nov 03 00:45:18.881904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.881823 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" event={"ID":"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac","Type":"ContainerStarted","Data":"4578bb6b41a86f4fdb5bbc873a54bd69ae572ac77a407eb018e50f1364c0f3f9"} Nov 03 00:45:18.881904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.881854 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:18.883441 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.883424 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" Nov 03 00:45:18.883541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.883452 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz" event={"ID":"c6b0e6fe-9364-465c-ad69-06216b67d22f","Type":"ContainerDied","Data":"429e556d0a143ab4df73b03088d9ec867b829b06a58e5c29b7ff3d08e4722812"} Nov 03 00:45:18.884847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.884819 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" Nov 03 00:45:18.884940 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.884847 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx" event={"ID":"4cd7204d-42da-41a9-8dc9-8b7892e05c69","Type":"ContainerDied","Data":"688025752ed9c51b84f92f3dd6f23d3f504be0a6b39b657014d8bc14361369b0"} Nov 03 00:45:18.895227 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.894601 5144 scope.go:122] "RemoveContainer" containerID="49068e885e7d4b07835341da3011c0c1bedb35d38f7fd4beaa508325b3cb4380" Nov 03 00:45:18.897738 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:18.897520 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96a426d_4339_4402_a7b4_fd6973106a08.slice/crio-d80eb0dc77440b6ec4d8445ebf2cf4763dd6091d9e7c577487d8b2e3fa7dab41 WatchSource:0}: Error finding container d80eb0dc77440b6ec4d8445ebf2cf4763dd6091d9e7c577487d8b2e3fa7dab41: Status 404 returned error can't find the container with id d80eb0dc77440b6ec4d8445ebf2cf4763dd6091d9e7c577487d8b2e3fa7dab41 Nov 03 00:45:18.899466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.899443 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp"] Nov 03 00:45:18.906081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.906065 5144 scope.go:122] "RemoveContainer" containerID="4f2b6946067a0e7221a7d15454e6cd33b88f80ca19a529f08a3282396f4a1cb1" Nov 03 00:45:18.940096 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.940063 5144 scope.go:122] "RemoveContainer" containerID="c1c093698bc87c7cb0138c19a676f4cfba389b13f696cb389ec1cb8e740bfb34" Nov 03 00:45:18.945462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:18.945302 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" podUID="861a520b-3348-4f6a-8fc6-ae574119b96e" containerName="main" containerID="cri-o://4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c" gracePeriod=45 Nov 03 00:45:19.205045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.205016 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:45:19.270436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.270361 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" podStartSLOduration=2.270299376 podStartE2EDuration="2.270299376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:19.250128243 +0000 UTC m=+2409.395822701" watchObservedRunningTime="2026-11-03 00:45:19.270299376 +0000 UTC m=+2409.415993831" Nov 03 00:45:19.313432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313400 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs\" (UniqueName: \"kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs\") pod \"861a520b-3348-4f6a-8fc6-ae574119b96e\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " Nov 03 00:45:19.313607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313443 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8\" (UniqueName: \"kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8\") pod \"861a520b-3348-4f6a-8fc6-ae574119b96e\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " Nov 03 00:45:19.313607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313532 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm\") pod \"861a520b-3348-4f6a-8fc6-ae574119b96e\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " Nov 03 00:45:19.313607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313586 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache\") pod \"861a520b-3348-4f6a-8fc6-ae574119b96e\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " Nov 03 00:45:19.313836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313680 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir\") pod \"861a520b-3348-4f6a-8fc6-ae574119b96e\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " Nov 03 00:45:19.313836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313706 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home\") pod \"861a520b-3348-4f6a-8fc6-ae574119b96e\" (UID: \"861a520b-3348-4f6a-8fc6-ae574119b96e\") " Nov 03 00:45:19.313939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313887 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache" pod "861a520b-3348-4f6a-8fc6-ae574119b96e" (UID: "861a520b-3348-4f6a-8fc6-ae574119b96e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:45:19.313939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.313919 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir" pod "861a520b-3348-4f6a-8fc6-ae574119b96e" (UID: "861a520b-3348-4f6a-8fc6-ae574119b96e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:45:19.314057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.314023 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home" pod "861a520b-3348-4f6a-8fc6-ae574119b96e" (UID: "861a520b-3348-4f6a-8fc6-ae574119b96e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:45:19.314337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.314265 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.314337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.314290 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.314337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.314304 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.316037 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.316006 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8" pod "861a520b-3348-4f6a-8fc6-ae574119b96e" (UID: "861a520b-3348-4f6a-8fc6-ae574119b96e"). InnerVolumeSpecName "kube-api-access-2kfq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:19.316383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.316362 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm" pod "861a520b-3348-4f6a-8fc6-ae574119b96e" (UID: "861a520b-3348-4f6a-8fc6-ae574119b96e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:45:19.316446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.316365 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs" pod "861a520b-3348-4f6a-8fc6-ae574119b96e" (UID: "861a520b-3348-4f6a-8fc6-ae574119b96e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:19.364417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.364360 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5"] Nov 03 00:45:19.372665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.372594 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894445-bd7v5"] Nov 03 00:45:19.415546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.415511 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/861a520b-3348-4f6a-8fc6-ae574119b96e-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.415546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.415547 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/861a520b-3348-4f6a-8fc6-ae574119b96e-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.415751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.415562 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kfq8\" (UniqueName: \"kubernetes.io/projected/861a520b-3348-4f6a-8fc6-ae574119b96e-kube-api-access-2kfq8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:19.477090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.476995 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx"] Nov 03 00:45:19.481501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.481473 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894445-jdfpx"] Nov 03 00:45:19.562023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.561979 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz"] Nov 03 00:45:19.590351 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.590321 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz"] Nov 03 00:45:19.595641 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.595602 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894445-x9skz"] Nov 03 00:45:19.897567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.897537 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" event={"ID":"e96a426d-4339-4402-a7b4-fd6973106a08","Type":"ContainerStarted","Data":"aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff"} Nov 03 00:45:19.898208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.897593 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" event={"ID":"e96a426d-4339-4402-a7b4-fd6973106a08","Type":"ContainerStarted","Data":"d80eb0dc77440b6ec4d8445ebf2cf4763dd6091d9e7c577487d8b2e3fa7dab41"} Nov 03 00:45:19.898208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.897618 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:19.899082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.899057 5144 generic.go:356] "Generic (PLEG): container finished" podID="861a520b-3348-4f6a-8fc6-ae574119b96e" containerID="4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c" exitCode=0 Nov 03 00:45:19.899214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.899121 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" Nov 03 00:45:19.899214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.899142 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" event={"ID":"861a520b-3348-4f6a-8fc6-ae574119b96e","Type":"ContainerDied","Data":"4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c"} Nov 03 00:45:19.899214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.899180 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd" event={"ID":"861a520b-3348-4f6a-8fc6-ae574119b96e","Type":"ContainerDied","Data":"f7e557edf833ff99607825a3a6646873aa5885a3ab330e38544a570bf0620aca"} Nov 03 00:45:19.899214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.899199 5144 scope.go:122] "RemoveContainer" containerID="4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c" Nov 03 00:45:19.913406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.913390 5144 scope.go:122] "RemoveContainer" containerID="4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c" Nov 03 00:45:19.913702 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:19.913677 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c\": container with ID starting with 4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c not found: ID does not exist" containerID="4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c" Nov 03 00:45:19.913800 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:19.913712 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c"} err="failed to get container status \"4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c\": rpc error: code = NotFound desc = could not find container \"4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c\": container with ID starting with 4b4d853b1fd0f163ca45e592da5d94de909b3fca479bb24c6ebb04df694e5f4c not found: ID does not exist" Nov 03 00:45:20.000172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.000144 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" path="/var/lib/kubelet/pods/4cd7204d-42da-41a9-8dc9-8b7892e05c69/volumes" Nov 03 00:45:20.000560 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.000546 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db2ef9f-1750-41ba-961c-cd866b63da96" path="/var/lib/kubelet/pods/4db2ef9f-1750-41ba-961c-cd866b63da96/volumes" Nov 03 00:45:20.000960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.000943 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" path="/var/lib/kubelet/pods/9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3/volumes" Nov 03 00:45:20.001678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.001594 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" path="/var/lib/kubelet/pods/c6b0e6fe-9364-465c-ad69-06216b67d22f/volumes" Nov 03 00:45:20.018936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.018890 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" podStartSLOduration=2.018874361 podStartE2EDuration="2.018874361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:20.009676014 +0000 UTC m=+2410.155370469" watchObservedRunningTime="2026-11-03 00:45:20.018874361 +0000 UTC m=+2410.164568818" Nov 03 00:45:20.067586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.067555 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:20.137015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.136909 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r\" (UniqueName: \"kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r\") pod \"64983805-ee5f-423d-9e9c-76a6f0dbea46\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " Nov 03 00:45:20.137341 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.137321 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca\") pod \"64983805-ee5f-423d-9e9c-76a6f0dbea46\" (UID: \"64983805-ee5f-423d-9e9c-76a6f0dbea46\") " Nov 03 00:45:20.137919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.137810 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca" pod "64983805-ee5f-423d-9e9c-76a6f0dbea46" (UID: "64983805-ee5f-423d-9e9c-76a6f0dbea46"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:20.138599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.138575 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64983805-ee5f-423d-9e9c-76a6f0dbea46-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:20.140212 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.140188 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r" pod "64983805-ee5f-423d-9e9c-76a6f0dbea46" (UID: "64983805-ee5f-423d-9e9c-76a6f0dbea46"). InnerVolumeSpecName "kube-api-access-vmz7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:20.240622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.240529 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmz7r\" (UniqueName: \"kubernetes.io/projected/64983805-ee5f-423d-9e9c-76a6f0dbea46-kube-api-access-vmz7r\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:20.248406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.248378 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd"] Nov 03 00:45:20.256768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.256737 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-d61eae/praxis-route-d61eae-kserve-76bccc49cc-svlfd"] Nov 03 00:45:20.914489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.914461 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" Nov 03 00:45:20.914935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.914492 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz" event={"ID":"64983805-ee5f-423d-9e9c-76a6f0dbea46","Type":"ContainerDied","Data":"18b72985e0e34dd8942d417118ca550fc04f99dafa73e04cd18db399d33c7a19"} Nov 03 00:45:20.914935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:20.914548 5144 scope.go:122] "RemoveContainer" containerID="afb83af1940a2c0d3d87d2a832b75cdeacb90d2e3d9695266611fb6ea5ef934e" Nov 03 00:45:21.078730 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:21.078703 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz"] Nov 03 00:45:21.084379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:21.084358 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-a-d61eae-29894445-c4cvz"] Nov 03 00:45:21.998254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:21.998217 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" path="/var/lib/kubelet/pods/64983805-ee5f-423d-9e9c-76a6f0dbea46/volumes" Nov 03 00:45:21.998844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:21.998824 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="861a520b-3348-4f6a-8fc6-ae574119b96e" path="/var/lib/kubelet/pods/861a520b-3348-4f6a-8fc6-ae574119b96e/volumes" Nov 03 00:45:29.905444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:29.905415 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 00:45:30.918508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:30.918474 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 00:45:34.276319 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:34.276286 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q"] Nov 03 00:45:34.384546 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:34.384509 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb"] Nov 03 00:45:35.193256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.193223 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc"] Nov 03 00:45:35.194099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194074 5144 cpu_manager.go:414] "containerMap: removing container" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.194099 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194097 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194110 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194117 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194132 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194139 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194178 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194186 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194195 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194202 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194220 5144 cpu_manager.go:414] "containerMap: removing container" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194226 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194239 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194245 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194282 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194290 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194306 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.194318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194313 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194343 5144 cpu_manager.go:414] "containerMap: removing container" podUID="861a520b-3348-4f6a-8fc6-ae574119b96e" containerName="main" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194352 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="861a520b-3348-4f6a-8fc6-ae574119b96e" containerName="main" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194547 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194566 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194577 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194587 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194598 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194608 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194621 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194650 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194661 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="861a520b-3348-4f6a-8fc6-ae574119b96e" containerName="main" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194672 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:35.195174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.194681 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:35.198745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.198724 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.251455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.250769 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:45:35.251455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.250943 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:45:35.254293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.254262 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc"] Nov 03 00:45:35.306462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.306436 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.306904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.306587 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsltm\" (UniqueName: \"kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.306904 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.306670 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.407848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.407816 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.408024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.407950 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsltm\" (UniqueName: \"kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.408024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.408007 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.408650 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.408607 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.410457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.410429 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.464346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.464278 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsltm\" (UniqueName: \"kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm\") pod \"maas-api-revoke-sub-59cda756-p87tc\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.513786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.513759 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:35.905131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:35.905100 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc"] Nov 03 00:45:35.906105 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:35.905768 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc022bb8d_cb70_4f60_9f68_b28b43e080c3.slice/crio-8f3ab32b5480ef1c3da021aaee0355bc1b6469c42f5c3254cf5fdfd3f00ff091 WatchSource:0}: Error finding container 8f3ab32b5480ef1c3da021aaee0355bc1b6469c42f5c3254cf5fdfd3f00ff091: Status 404 returned error can't find the container with id 8f3ab32b5480ef1c3da021aaee0355bc1b6469c42f5c3254cf5fdfd3f00ff091 Nov 03 00:45:36.051703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:36.051429 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" event={"ID":"c022bb8d-cb70-4f60-9f68-b28b43e080c3","Type":"ContainerStarted","Data":"8f3ab32b5480ef1c3da021aaee0355bc1b6469c42f5c3254cf5fdfd3f00ff091"} Nov 03 00:45:37.061000 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:37.060968 5144 generic.go:356] "Generic (PLEG): container finished" podID="c022bb8d-cb70-4f60-9f68-b28b43e080c3" containerID="5a9e7a7202965ee91513d6ef96e09940be808b8ee6599e840bbc5457fbf05fb3" exitCode=0 Nov 03 00:45:37.061346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:37.061056 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" event={"ID":"c022bb8d-cb70-4f60-9f68-b28b43e080c3","Type":"ContainerDied","Data":"5a9e7a7202965ee91513d6ef96e09940be808b8ee6599e840bbc5457fbf05fb3"} Nov 03 00:45:38.226300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.226277 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:38.351830 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.351762 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca\") pod \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " Nov 03 00:45:38.351988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.351853 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca\") pod \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " Nov 03 00:45:38.351988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.351931 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm\" (UniqueName: \"kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm\") pod \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\" (UID: \"c022bb8d-cb70-4f60-9f68-b28b43e080c3\") " Nov 03 00:45:38.352283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.352253 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca" pod "c022bb8d-cb70-4f60-9f68-b28b43e080c3" (UID: "c022bb8d-cb70-4f60-9f68-b28b43e080c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:38.352573 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.352548 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:38.354802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.354776 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm" pod "c022bb8d-cb70-4f60-9f68-b28b43e080c3" (UID: "c022bb8d-cb70-4f60-9f68-b28b43e080c3"). InnerVolumeSpecName "kube-api-access-wsltm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:38.357764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.357725 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca" pod "c022bb8d-cb70-4f60-9f68-b28b43e080c3" (UID: "c022bb8d-cb70-4f60-9f68-b28b43e080c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:45:38.454177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.454150 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsltm\" (UniqueName: \"kubernetes.io/projected/c022bb8d-cb70-4f60-9f68-b28b43e080c3-kube-api-access-wsltm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:38.454177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:38.454175 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c022bb8d-cb70-4f60-9f68-b28b43e080c3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:39.077216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:39.077188 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" Nov 03 00:45:39.077216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:39.077197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc" event={"ID":"c022bb8d-cb70-4f60-9f68-b28b43e080c3","Type":"ContainerDied","Data":"8f3ab32b5480ef1c3da021aaee0355bc1b6469c42f5c3254cf5fdfd3f00ff091"} Nov 03 00:45:39.077450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:39.077254 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f3ab32b5480ef1c3da021aaee0355bc1b6469c42f5c3254cf5fdfd3f00ff091" Nov 03 00:45:39.276728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:39.276686 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" podUID="b5c23b68-630d-483f-9720-4070d6743874" containerName="payload-processing" containerID="cri-o://b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654" gracePeriod=25 Nov 03 00:45:39.385165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:39.385072 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerName="payload-pre-processing" containerID="cri-o://7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b" gracePeriod=25 Nov 03 00:45:42.156060 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:45:42.156013 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:45:42.617369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:42.617334 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:45:43.624503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.624463 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" podUID="b5c23b68-630d-483f-9720-4070d6743874" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:45:43.650221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650185 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr"] Nov 03 00:45:43.650908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650887 5144 cpu_manager.go:414] "containerMap: removing container" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650908 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64983805-ee5f-423d-9e9c-76a6f0dbea46" containerName="cleanup" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650934 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c022bb8d-cb70-4f60-9f68-b28b43e080c3" containerName="revoke-keys" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650942 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c022bb8d-cb70-4f60-9f68-b28b43e080c3" containerName="revoke-keys" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650990 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.650997 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.651009 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:43.651020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.651017 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6b0e6fe-9364-465c-ad69-06216b67d22f" containerName="cleanup" Nov 03 00:45:43.651392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.651186 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd7204d-42da-41a9-8dc9-8b7892e05c69" containerName="cleanup" Nov 03 00:45:43.651392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.651205 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8fa8b3-4ee9-4a97-8ae2-f49c77a3b9a3" containerName="cleanup" Nov 03 00:45:43.651392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.651222 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c022bb8d-cb70-4f60-9f68-b28b43e080c3" containerName="revoke-keys" Nov 03 00:45:43.656172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.656151 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.756119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.756086 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr"] Nov 03 00:45:43.816837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.816800 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkglr\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.816863 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.816971 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.817004 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.817029 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.817121 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.817158 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.817268 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.817385 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.817335 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918428 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918512 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkglr\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918540 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918567 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918589 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918614 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918675 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.918707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918699 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.919125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918738 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.919125 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.918913 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.919315 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.919290 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.919504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.919481 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.919696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.919675 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.919772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.919701 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.921240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.921215 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.922006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.921986 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.974484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.974463 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:43.974582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:43.974525 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkglr\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:44.270685 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:44.270654 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:44.705452 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:45:44.705422 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70eced92_3511_4c69_9b83_d599f3e0f449.slice/crio-7abd0354be24df5978f82ac368e35ef3469f093120eb492afb897b781b0a4092 WatchSource:0}: Error finding container 7abd0354be24df5978f82ac368e35ef3469f093120eb492afb897b781b0a4092: Status 404 returned error can't find the container with id 7abd0354be24df5978f82ac368e35ef3469f093120eb492afb897b781b0a4092 Nov 03 00:45:44.705911 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:44.705851 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr"] Nov 03 00:45:44.707794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:44.707765 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:44.708036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:44.707827 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:44.708036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:44.707855 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:45.131645 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:45.131599 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" event={"ID":"70eced92-3511-4c69-9b83-d599f3e0f449","Type":"ContainerStarted","Data":"eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84"} Nov 03 00:45:45.131857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:45.131694 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" event={"ID":"70eced92-3511-4c69-9b83-d599f3e0f449","Type":"ContainerStarted","Data":"7abd0354be24df5978f82ac368e35ef3469f093120eb492afb897b781b0a4092"} Nov 03 00:45:45.252033 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:45.251971 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podStartSLOduration=2.251953722 podStartE2EDuration="2.251953722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:45.247978926 +0000 UTC m=+2435.393673392" watchObservedRunningTime="2026-11-03 00:45:45.251953722 +0000 UTC m=+2435.397648178" Nov 03 00:45:45.270896 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:45.270861 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:45.272715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:45.272688 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:45.272864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:45.272749 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:46.271374 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:46.271336 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:46.271819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:46.271397 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:47.271260 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:47.271221 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:47.271458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:47.271289 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:48.271083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:48.271044 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:48.272045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:48.272012 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:49.271063 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:49.271029 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:49.271326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:49.271292 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:50.272379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:50.271188 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:50.272379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:50.271239 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:51.271311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:51.271280 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" start-of-body= Nov 03 00:45:51.271510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:51.271331 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.237:15021/healthz/ready\": dial tcp 10.132.0.237:15021: connect: connection refused" Nov 03 00:45:52.392105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:52.392076 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:52.402935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:52.402873 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:52.407481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:52.407448 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:45:52.633656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:52.632712 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:45:53.624738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:53.624687 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" podUID="b5c23b68-630d-483f-9720-4070d6743874" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 00:45:57.319459 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:57.319418 5144 generic.go:356] "Generic (PLEG): container finished" podID="5bf746e0-37b0-4325-b5b7-31465218f238" containerID="0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22" exitCode=0 Nov 03 00:45:57.319971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:57.319495 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" event={"ID":"5bf746e0-37b0-4325-b5b7-31465218f238","Type":"ContainerDied","Data":"0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22"} Nov 03 00:45:57.319971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:57.319957 5144 scope.go:122] "RemoveContainer" containerID="0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22" Nov 03 00:45:57.321502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:57.321469 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:57.321661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:57.321538 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:57.321661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:57.321571 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:45:58.347222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:58.347188 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" event={"ID":"5bf746e0-37b0-4325-b5b7-31465218f238","Type":"ContainerStarted","Data":"c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8"} Nov 03 00:45:59.163463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.163429 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:45:59.163712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.163685 5144 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 00:45:59.163765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.163747 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 00:45:59.588997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.588972 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:45:59.696895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.696874 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:45:59.697220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.697207 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume\") pod \"b5c23b68-630d-483f-9720-4070d6743874\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " Nov 03 00:45:59.697262 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.697244 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx\" (UniqueName: \"kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx\") pod \"b5c23b68-630d-483f-9720-4070d6743874\" (UID: \"b5c23b68-630d-483f-9720-4070d6743874\") " Nov 03 00:45:59.697555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.697534 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume" pod "b5c23b68-630d-483f-9720-4070d6743874" (UID: "b5c23b68-630d-483f-9720-4070d6743874"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:59.697797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.697782 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5c23b68-630d-483f-9720-4070d6743874-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:59.699379 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.699361 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx" pod "b5c23b68-630d-483f-9720-4070d6743874" (UID: "b5c23b68-630d-483f-9720-4070d6743874"). InnerVolumeSpecName "kube-api-access-74bfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:59.799256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.799229 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume\") pod \"01a4cf82-5705-4290-8e72-80cf9985e6a0\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " Nov 03 00:45:59.799439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.799272 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq\" (UniqueName: \"kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq\") pod \"01a4cf82-5705-4290-8e72-80cf9985e6a0\" (UID: \"01a4cf82-5705-4290-8e72-80cf9985e6a0\") " Nov 03 00:45:59.799588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.799568 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume" pod "01a4cf82-5705-4290-8e72-80cf9985e6a0" (UID: "01a4cf82-5705-4290-8e72-80cf9985e6a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:45:59.799765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.799741 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01a4cf82-5705-4290-8e72-80cf9985e6a0-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:59.799765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.799760 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74bfx\" (UniqueName: \"kubernetes.io/projected/b5c23b68-630d-483f-9720-4070d6743874-kube-api-access-74bfx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:45:59.801506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.801483 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq" pod "01a4cf82-5705-4290-8e72-80cf9985e6a0" (UID: "01a4cf82-5705-4290-8e72-80cf9985e6a0"). InnerVolumeSpecName "kube-api-access-m7ndq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:45:59.901186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:45:59.901150 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7ndq\" (UniqueName: \"kubernetes.io/projected/01a4cf82-5705-4290-8e72-80cf9985e6a0-kube-api-access-m7ndq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:46:00.163682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.163578 5144 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 00:46:00.163841 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.163676 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 00:46:00.364228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.364198 5144 generic.go:356] "Generic (PLEG): container finished" podID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerID="7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b" exitCode=0 Nov 03 00:46:00.364406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.364245 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" Nov 03 00:46:00.364406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.364293 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" event={"ID":"01a4cf82-5705-4290-8e72-80cf9985e6a0","Type":"ContainerDied","Data":"7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b"} Nov 03 00:46:00.364406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.364345 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb" event={"ID":"01a4cf82-5705-4290-8e72-80cf9985e6a0","Type":"ContainerDied","Data":"ad3f71b0c11b638d3bdcff5e658d36a3a79fa4c955f1f5e1ffe37f8bafb25e2b"} Nov 03 00:46:00.364406 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.364376 5144 scope.go:122] "RemoveContainer" containerID="7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b" Nov 03 00:46:00.365902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.365878 5144 generic.go:356] "Generic (PLEG): container finished" podID="b5c23b68-630d-483f-9720-4070d6743874" containerID="b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654" exitCode=0 Nov 03 00:46:00.366030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.365926 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" Nov 03 00:46:00.366030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.366011 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" event={"ID":"b5c23b68-630d-483f-9720-4070d6743874","Type":"ContainerDied","Data":"b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654"} Nov 03 00:46:00.366140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.366057 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q" event={"ID":"b5c23b68-630d-483f-9720-4070d6743874","Type":"ContainerDied","Data":"4aaf03d0c57aac89ba616d9be237dc4c0373bf6a968709711c314fefca5bc010"} Nov 03 00:46:00.375734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.375715 5144 scope.go:122] "RemoveContainer" containerID="7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b" Nov 03 00:46:00.376016 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:46:00.375988 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b\": container with ID starting with 7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b not found: ID does not exist" containerID="7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b" Nov 03 00:46:00.376080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.376014 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b"} err="failed to get container status \"7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b\": rpc error: code = NotFound desc = could not find container \"7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b\": container with ID starting with 7c8d3932b2377afc502c25eb509aa6ac7c3adde848571fe5d2491e4312b3820b not found: ID does not exist" Nov 03 00:46:00.376080 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.376034 5144 scope.go:122] "RemoveContainer" containerID="b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654" Nov 03 00:46:00.389997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.389974 5144 scope.go:122] "RemoveContainer" containerID="b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654" Nov 03 00:46:00.390327 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:46:00.390301 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654\": container with ID starting with b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654 not found: ID does not exist" containerID="b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654" Nov 03 00:46:00.390443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.390332 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654"} err="failed to get container status \"b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654\": rpc error: code = NotFound desc = could not find container \"b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654\": container with ID starting with b6a7ad5ab94be2daea97a071d74b15db8702447934a5c018dd602a897994c654 not found: ID does not exist" Nov 03 00:46:00.545450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.545421 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q"] Nov 03 00:46:00.549768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.549745 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-d61eae-7ffb7f659-xh27q"] Nov 03 00:46:00.652283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.652253 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb"] Nov 03 00:46:00.656439 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:00.656415 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-d61eae-7877fd8d55-fkktb"] Nov 03 00:46:01.162721 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:01.162680 5144 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 00:46:01.162991 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:01.162745 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 00:46:01.998338 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:01.998301 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" path="/var/lib/kubelet/pods/01a4cf82-5705-4290-8e72-80cf9985e6a0/volumes" Nov 03 00:46:01.999371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:01.998946 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5c23b68-630d-483f-9720-4070d6743874" path="/var/lib/kubelet/pods/b5c23b68-630d-483f-9720-4070d6743874/volumes" Nov 03 00:46:02.163242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:02.163207 5144 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 00:46:02.163420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:02.163256 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 00:46:03.163431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:03.163385 5144 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" start-of-body= Nov 03 00:46:03.163874 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:03.163457 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.221:15021/healthz/ready\": dial tcp 10.132.0.221:15021: connect: connection refused" Nov 03 00:46:04.167581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:04.167548 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:46:04.168081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:04.167912 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:46:04.168510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:04.168495 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 00:46:07.047231 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:07.047196 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh"] Nov 03 00:46:07.097062 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:07.097037 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859ed6fe-gldvh"] Nov 03 00:46:07.996688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:07.996655 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97f0b73c-20cc-45bd-8bdc-cce59b1619f1" path="/var/lib/kubelet/pods/97f0b73c-20cc-45bd-8bdc-cce59b1619f1/volumes" Nov 03 00:46:12.157562 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:46:12.157529 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:46:26.501209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.501127 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5"] Nov 03 00:46:26.501971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.501949 5144 cpu_manager.go:414] "containerMap: removing container" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerName="payload-pre-processing" Nov 03 00:46:26.502089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.501971 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerName="payload-pre-processing" Nov 03 00:46:26.502089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.502037 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b5c23b68-630d-483f-9720-4070d6743874" containerName="payload-processing" Nov 03 00:46:26.502089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.502045 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5c23b68-630d-483f-9720-4070d6743874" containerName="payload-processing" Nov 03 00:46:26.502247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.502207 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5c23b68-630d-483f-9720-4070d6743874" containerName="payload-processing" Nov 03 00:46:26.502247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.502224 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="01a4cf82-5705-4290-8e72-80cf9985e6a0" containerName="payload-pre-processing" Nov 03 00:46:26.505959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.505937 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.559066 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.559029 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-904a34-openshift-default-dockercfg-dww4p\"" Nov 03 00:46:26.619970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.619936 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5"] Nov 03 00:46:26.650709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.650667 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.651134 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.651110 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.651336 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.651317 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.651466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.651448 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8ddg\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.656171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.651619 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.656334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.656218 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.656334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.656276 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.656334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.656312 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.656510 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.656347 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.757487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757457 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.757758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757504 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.757758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757653 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.757758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757707 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.757758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757838 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757921 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757944 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.757982 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.758014 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8ddg\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.758441 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758536 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.758487 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.758668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.758649 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.759140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.759119 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.761647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.761604 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.761915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.761894 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.816666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.816603 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.816776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.816665 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8ddg\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg\") pod \"e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:26.822539 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:26.822520 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:27.263068 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:46:27.263039 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6052680c_30b2_44f4_ac41_f24f33dc8fca.slice/crio-545e307866bd9b7214ec3a22d129e4705d530068ea2531906eccfbc30f3e55be WatchSource:0}: Error finding container 545e307866bd9b7214ec3a22d129e4705d530068ea2531906eccfbc30f3e55be: Status 404 returned error can't find the container with id 545e307866bd9b7214ec3a22d129e4705d530068ea2531906eccfbc30f3e55be Nov 03 00:46:27.263481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.263452 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5"] Nov 03 00:46:27.265409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.265380 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:27.265483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.265440 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:27.265483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.265469 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:27.612596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.612515 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" event={"ID":"6052680c-30b2-44f4-ac41-f24f33dc8fca","Type":"ContainerStarted","Data":"a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03"} Nov 03 00:46:27.612596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.612571 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" event={"ID":"6052680c-30b2-44f4-ac41-f24f33dc8fca","Type":"ContainerStarted","Data":"545e307866bd9b7214ec3a22d129e4705d530068ea2531906eccfbc30f3e55be"} Nov 03 00:46:27.731172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.731123 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podStartSLOduration=1.731109467 podStartE2EDuration="1.731109467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:27.728412911 +0000 UTC m=+2477.874107366" watchObservedRunningTime="2026-11-03 00:46:27.731109467 +0000 UTC m=+2477.876803923" Nov 03 00:46:27.823437 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:27.823411 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:28.829386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:28.829335 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:29.628055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:29.628023 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:29.629130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:29.629103 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:46:34.697356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.697317 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd"] Nov 03 00:46:34.704468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.704441 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.753872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.753843 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.753922 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhvp8\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.753998 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.754134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.754162 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.754213 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.754259 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.754311 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.754438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.754340 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.761549 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.761519 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-328e1b-openshift-default-dockercfg-rgsh9\"" Nov 03 00:46:34.818652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.815987 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd"] Nov 03 00:46:34.856514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.856470 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhvp8\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.856735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.856566 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.856989 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.857028 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.857077 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.857117 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.857165 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857236 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.857191 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.857519 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.857256 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.858145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.858111 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.858755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.858734 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.859361 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.859309 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.859585 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.859561 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.860054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.860031 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.874340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.874306 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.890141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.890110 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.916211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.916179 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhvp8\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:34.916963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:34.916943 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token\") pod \"e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:35.037380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.037347 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:35.482150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.482073 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd"] Nov 03 00:46:35.517119 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:46:35.517090 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46ed093_2590_4248_ad19_409dd68099a1.slice/crio-856c2694da6cd8e24825baf918959c49ddf4234a462503f83a692dc8fbcc7090 WatchSource:0}: Error finding container 856c2694da6cd8e24825baf918959c49ddf4234a462503f83a692dc8fbcc7090: Status 404 returned error can't find the container with id 856c2694da6cd8e24825baf918959c49ddf4234a462503f83a692dc8fbcc7090 Nov 03 00:46:35.520196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.520164 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:35.520287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.520249 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:35.520287 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.520279 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:46:35.678524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.678492 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" event={"ID":"f46ed093-2590-4248-ad19-409dd68099a1","Type":"ContainerStarted","Data":"75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93"} Nov 03 00:46:35.678714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.678544 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" event={"ID":"f46ed093-2590-4248-ad19-409dd68099a1","Type":"ContainerStarted","Data":"856c2694da6cd8e24825baf918959c49ddf4234a462503f83a692dc8fbcc7090"} Nov 03 00:46:35.797394 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:35.797345 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podStartSLOduration=1.7973305929999999 podStartE2EDuration="1.797330593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:35.793421573 +0000 UTC m=+2485.939116033" watchObservedRunningTime="2026-11-03 00:46:35.797330593 +0000 UTC m=+2485.943025048" Nov 03 00:46:36.038119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:36.038089 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:36.043776 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:36.043751 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:36.687575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:36.687541 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:36.688690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:36.688670 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:46:37.869193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:37.869159 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq"] Nov 03 00:46:37.873715 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:37.873690 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:37.902942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:37.902906 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddrmh\" (UniqueName: \"kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh\") pod \"payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:37.903106 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:37.903073 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:37.931584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:37.931553 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq"] Nov 03 00:46:37.986068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:37.986028 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-904a34-dockercfg-b8zxr\"" Nov 03 00:46:38.005998 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.005960 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-904a34\"" Nov 03 00:46:38.036658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.015320 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddrmh\" (UniqueName: \"kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh\") pod \"payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:38.036658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.015478 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:38.036658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.016489 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:38.067082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.067042 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd"] Nov 03 00:46:38.071835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.071812 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.090736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.088135 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd"] Nov 03 00:46:38.123724 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.122230 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddrmh\" (UniqueName: \"kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh\") pod \"payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:38.190038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.189993 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:38.218162 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.218133 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94r89\" (UniqueName: \"kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89\") pod \"payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.218514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.218487 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.320223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.320177 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.320418 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.320260 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94r89\" (UniqueName: \"kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89\") pod \"payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.321014 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.320985 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.417707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.411189 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r89\" (UniqueName: \"kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89\") pod \"payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.575333 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:46:38.575295 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d4c4328_fc5b_4eba_afad_ac674b898816.slice/crio-0a7975b674f3ae3ab53e0ee96763408d8d2901b6a21c735300485ad4dc8befab WatchSource:0}: Error finding container 0a7975b674f3ae3ab53e0ee96763408d8d2901b6a21c735300485ad4dc8befab: Status 404 returned error can't find the container with id 0a7975b674f3ae3ab53e0ee96763408d8d2901b6a21c735300485ad4dc8befab Nov 03 00:46:38.576922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.576892 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq"] Nov 03 00:46:38.690600 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.690546 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:38.706314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.706285 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" event={"ID":"7d4c4328-fc5b-4eba-afad-ac674b898816","Type":"ContainerStarted","Data":"82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333"} Nov 03 00:46:38.706426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.706336 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" event={"ID":"7d4c4328-fc5b-4eba-afad-ac674b898816","Type":"ContainerStarted","Data":"0a7975b674f3ae3ab53e0ee96763408d8d2901b6a21c735300485ad4dc8befab"} Nov 03 00:46:38.825090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:38.825040 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" podStartSLOduration=1.82502765 podStartE2EDuration="1.82502765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:38.821789421 +0000 UTC m=+2488.967483895" watchObservedRunningTime="2026-11-03 00:46:38.82502765 +0000 UTC m=+2488.970722097" Nov 03 00:46:39.069156 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:46:39.069128 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb25748e5_b0fa_47a4_948a_170d3ef2f6e9.slice/crio-29b0472d198724f01ecf632ec4977b1c21766e1120aa2ae2cafb1b8374fe84fd WatchSource:0}: Error finding container 29b0472d198724f01ecf632ec4977b1c21766e1120aa2ae2cafb1b8374fe84fd: Status 404 returned error can't find the container with id 29b0472d198724f01ecf632ec4977b1c21766e1120aa2ae2cafb1b8374fe84fd Nov 03 00:46:39.069591 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:39.069490 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd"] Nov 03 00:46:39.717401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:39.717366 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" event={"ID":"b25748e5-b0fa-47a4-948a-170d3ef2f6e9","Type":"ContainerStarted","Data":"d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf"} Nov 03 00:46:39.717595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:39.717449 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" event={"ID":"b25748e5-b0fa-47a4-948a-170d3ef2f6e9","Type":"ContainerStarted","Data":"29b0472d198724f01ecf632ec4977b1c21766e1120aa2ae2cafb1b8374fe84fd"} Nov 03 00:46:39.717595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:39.717475 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:39.717861 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:39.717840 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:39.831372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:39.831322 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" podStartSLOduration=2.831309127 podStartE2EDuration="2.831309127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:39.830070444 +0000 UTC m=+2489.975764899" watchObservedRunningTime="2026-11-03 00:46:39.831309127 +0000 UTC m=+2489.977003616" Nov 03 00:46:42.155174 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:46:42.155132 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:46:42.781045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.781016 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7"] Nov 03 00:46:42.785876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.785855 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:42.841902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.841874 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7"] Nov 03 00:46:42.886916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.886875 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-328e1b-dockercfg-6qdxl\"" Nov 03 00:46:42.887126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.887009 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-328e1b\"" Nov 03 00:46:42.903653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.900134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47bm\" (UniqueName: \"kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm\") pod \"payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:42.903653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.900215 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:42.950684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.950616 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg"] Nov 03 00:46:42.956363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.956339 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:42.988370 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:42.988340 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg"] Nov 03 00:46:43.000865 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.000829 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.001002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.000870 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47bm\" (UniqueName: \"kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm\") pod \"payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:43.001002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.000966 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4kvh\" (UniqueName: \"kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh\") pod \"payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.001092 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.001037 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:43.001813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.001791 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:43.057416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.057350 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47bm\" (UniqueName: \"kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm\") pod \"payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:43.101036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.101012 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:43.101611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.101590 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4kvh\" (UniqueName: \"kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh\") pod \"payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.101806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.101790 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.102398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.102373 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.160913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.160833 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4kvh\" (UniqueName: \"kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh\") pod \"payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.271929 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.271896 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.479571 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:46:43.479540 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5e6d74_afac_4955_9634_c6f6374d0bd3.slice/crio-2820b2fc896ad8c7a6fc8f97dbbde717366c687beda7103eca7a3cf3957590a7 WatchSource:0}: Error finding container 2820b2fc896ad8c7a6fc8f97dbbde717366c687beda7103eca7a3cf3957590a7: Status 404 returned error can't find the container with id 2820b2fc896ad8c7a6fc8f97dbbde717366c687beda7103eca7a3cf3957590a7 Nov 03 00:46:43.480362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.480334 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7"] Nov 03 00:46:43.650508 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:46:43.650475 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c39946_84c5_42f5_9dc4_ca66cdebea2b.slice/crio-b60ef14f1751e35015cf0ac89884ce96dc82a7074591589bcf3ef1fa0ccefaf4 WatchSource:0}: Error finding container b60ef14f1751e35015cf0ac89884ce96dc82a7074591589bcf3ef1fa0ccefaf4: Status 404 returned error can't find the container with id b60ef14f1751e35015cf0ac89884ce96dc82a7074591589bcf3ef1fa0ccefaf4 Nov 03 00:46:43.651201 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.651180 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg"] Nov 03 00:46:43.750463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.750429 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" event={"ID":"5b5e6d74-afac-4955-9634-c6f6374d0bd3","Type":"ContainerStarted","Data":"03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3"} Nov 03 00:46:43.750586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.750491 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" event={"ID":"5b5e6d74-afac-4955-9634-c6f6374d0bd3","Type":"ContainerStarted","Data":"2820b2fc896ad8c7a6fc8f97dbbde717366c687beda7103eca7a3cf3957590a7"} Nov 03 00:46:43.750586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.750516 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:43.752200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.752175 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" event={"ID":"05c39946-84c5-42f5-9dc4-ca66cdebea2b","Type":"ContainerStarted","Data":"54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e"} Nov 03 00:46:43.752299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.752224 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:46:43.752299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.752245 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" event={"ID":"05c39946-84c5-42f5-9dc4-ca66cdebea2b","Type":"ContainerStarted","Data":"b60ef14f1751e35015cf0ac89884ce96dc82a7074591589bcf3ef1fa0ccefaf4"} Nov 03 00:46:43.864514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.864463 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" podStartSLOduration=1.864450596 podStartE2EDuration="1.864450596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:43.861747737 +0000 UTC m=+2494.007442211" watchObservedRunningTime="2026-11-03 00:46:43.864450596 +0000 UTC m=+2494.010145051" Nov 03 00:46:43.982303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:43.982190 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" podStartSLOduration=1.982174571 podStartE2EDuration="1.982174571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:43.973371173 +0000 UTC m=+2494.119065648" watchObservedRunningTime="2026-11-03 00:46:43.982174571 +0000 UTC m=+2494.127869060" Nov 03 00:46:50.728412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:50.728380 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 00:46:50.729021 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:50.728465 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 00:46:54.765576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:54.765543 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 00:46:54.766056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:46:54.765621 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 00:47:03.049417 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:03.049384 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd"] Nov 03 00:47:03.101368 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:03.101332 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33286da1-v5vmd"] Nov 03 00:47:03.996759 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:03.996728 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cd1bf3e-b3b4-48a3-815d-8330dae28947" path="/var/lib/kubelet/pods/9cd1bf3e-b3b4-48a3-815d-8330dae28947/volumes" Nov 03 00:47:06.078455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.078422 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9"] Nov 03 00:47:06.084017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.083994 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.139464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.139436 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9"] Nov 03 00:47:06.186816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.186028 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c87e2f\"" Nov 03 00:47:06.186816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.186164 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c87e2f-dockercfg-2bb87\"" Nov 03 00:47:06.211402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.201991 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr74z\" (UniqueName: \"kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z\") pod \"payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.225699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.225659 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.226045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.224339 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8"] Nov 03 00:47:06.231601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.231579 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.293531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.293501 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8"] Nov 03 00:47:06.327658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.327603 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.327969 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.327948 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr74z\" (UniqueName: \"kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z\") pod \"payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.328110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.328093 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.328310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.328290 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rpws\" (UniqueName: \"kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws\") pod \"payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.329234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.329167 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.385146 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.385115 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr74z\" (UniqueName: \"kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z\") pod \"payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.400121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.400096 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:06.430036 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.430004 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.432372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.431978 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rpws\" (UniqueName: \"kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws\") pod \"payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.432372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.431861 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.494867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.494807 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rpws\" (UniqueName: \"kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws\") pod \"payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.593495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.593407 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:06.787527 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:47:06.787493 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66b80d61_39d5_465d_a156_45fcd3ad0722.slice/crio-aa008ceec0018dd7ec164ec8eb81273a71c295f13fc6773d8e70c2b314e5eb75 WatchSource:0}: Error finding container aa008ceec0018dd7ec164ec8eb81273a71c295f13fc6773d8e70c2b314e5eb75: Status 404 returned error can't find the container with id aa008ceec0018dd7ec164ec8eb81273a71c295f13fc6773d8e70c2b314e5eb75 Nov 03 00:47:06.787928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.787903 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9"] Nov 03 00:47:06.971854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.971815 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" event={"ID":"66b80d61-39d5-465d-a156-45fcd3ad0722","Type":"ContainerStarted","Data":"aa008ceec0018dd7ec164ec8eb81273a71c295f13fc6773d8e70c2b314e5eb75"} Nov 03 00:47:06.997614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:06.997439 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8"] Nov 03 00:47:07.987329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:07.987294 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" event={"ID":"9a14e39a-7816-4289-af4e-a95614befbf8","Type":"ContainerStarted","Data":"0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5"} Nov 03 00:47:07.987852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:07.987355 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:07.987852 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:07.987380 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" event={"ID":"9a14e39a-7816-4289-af4e-a95614befbf8","Type":"ContainerStarted","Data":"2981a2b9ed39c609bb797b6add9318bf65d03348f76c1574d33399aa2ba54538"} Nov 03 00:47:07.996366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:07.996339 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" event={"ID":"66b80d61-39d5-465d-a156-45fcd3ad0722","Type":"ContainerStarted","Data":"83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972"} Nov 03 00:47:07.996503 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:07.996388 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:08.102163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:08.102123 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" podStartSLOduration=2.102111747 podStartE2EDuration="2.102111747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:08.100363576 +0000 UTC m=+2518.246058025" watchObservedRunningTime="2026-11-03 00:47:08.102111747 +0000 UTC m=+2518.247806203" Nov 03 00:47:08.216458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:08.216374 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" podStartSLOduration=2.216355981 podStartE2EDuration="2.216355981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:08.21331175 +0000 UTC m=+2518.359006225" watchObservedRunningTime="2026-11-03 00:47:08.216355981 +0000 UTC m=+2518.362050438" Nov 03 00:47:12.155018 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:47:12.154980 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:47:12.157877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:12.157855 5144 scope.go:122] "RemoveContainer" containerID="ad994fd18c57538bb4616485f8960d7a64440fa7543851d132a9f41391ee6e8d" Nov 03 00:47:16.057237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:16.057193 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2"] Nov 03 00:47:16.100739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:16.100703 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edd15c04-cjrs2"] Nov 03 00:47:17.960624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:17.960593 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf"] Nov 03 00:47:17.961139 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:17.960837 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" containerID="cri-o://1218918e642ed21cf1a77ac981e06a82506e3591a6119c485a56326ef3c20d96" gracePeriod=30 Nov 03 00:47:17.998009 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:17.997976 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9864fb78-6fa7-4c02-b38c-695fc3c65bbc" path="/var/lib/kubelet/pods/9864fb78-6fa7-4c02-b38c-695fc3c65bbc/volumes" Nov 03 00:47:19.020022 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:19.019995 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 00:47:19.020413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:19.020067 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 00:47:23.137999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.137963 5144 generic.go:356] "Generic (PLEG): container finished" podID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerID="1218918e642ed21cf1a77ac981e06a82506e3591a6119c485a56326ef3c20d96" exitCode=0 Nov 03 00:47:23.138429 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.138022 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" event={"ID":"de5382a4-91d6-44f4-8911-dfac0b6d7382","Type":"ContainerDied","Data":"1218918e642ed21cf1a77ac981e06a82506e3591a6119c485a56326ef3c20d96"} Nov 03 00:47:23.138429 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.138107 5144 scope.go:122] "RemoveContainer" containerID="511837c54d27ba68078cceaea432913786c022cf048bd9858d7f55a345aebe44" Nov 03 00:47:23.239902 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.239842 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:47:23.256509 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256486 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256548 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256601 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256655 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256685 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256748 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256788 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256831 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.256930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256786 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:23.257136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256950 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:23.257136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256903 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:23.257136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.256994 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo\") pod \"de5382a4-91d6-44f4-8911-dfac0b6d7382\" (UID: \"de5382a4-91d6-44f4-8911-dfac0b6d7382\") " Nov 03 00:47:23.257136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257091 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:23.257357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257243 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:23.257553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257534 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.257654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257557 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.257654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257571 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de5382a4-91d6-44f4-8911-dfac0b6d7382-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.257654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257583 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.257654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.257597 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.258829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.258808 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:47:23.259024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.259004 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "kube-api-access-dx2s6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:23.259440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.259421 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:23.259521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.259503 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo" pod "de5382a4-91d6-44f4-8911-dfac0b6d7382" (UID: "de5382a4-91d6-44f4-8911-dfac0b6d7382"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:47:23.358218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.358196 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.358218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.358219 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.358359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.358229 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx2s6\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-kube-api-access-dx2s6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:23.358359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:23.358238 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de5382a4-91d6-44f4-8911-dfac0b6d7382-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:24.147342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:24.147312 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" Nov 03 00:47:24.147825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:24.147334 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf" event={"ID":"de5382a4-91d6-44f4-8911-dfac0b6d7382","Type":"ContainerDied","Data":"409da360ca765181e0922fd2d5a843536219e654002517fe3ecac9063a04ed5d"} Nov 03 00:47:24.147825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:24.147398 5144 scope.go:122] "RemoveContainer" containerID="1218918e642ed21cf1a77ac981e06a82506e3591a6119c485a56326ef3c20d96" Nov 03 00:47:24.365403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:24.365319 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf"] Nov 03 00:47:24.369802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:24.369775 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bdd194-openshift-default-85c9569fb9-74wjf"] Nov 03 00:47:25.997275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:25.997243 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" path="/var/lib/kubelet/pods/de5382a4-91d6-44f4-8911-dfac0b6d7382/volumes" Nov 03 00:47:31.239222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.239182 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6"] Nov 03 00:47:31.239854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.239838 5144 cpu_manager.go:414] "containerMap: removing container" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" Nov 03 00:47:31.239854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.239854 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" Nov 03 00:47:31.239982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.239876 5144 cpu_manager.go:414] "containerMap: removing container" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" Nov 03 00:47:31.239982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.239882 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" Nov 03 00:47:31.240093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.240001 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" Nov 03 00:47:31.240093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.240012 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="de5382a4-91d6-44f4-8911-dfac0b6d7382" containerName="istio-proxy" Nov 03 00:47:31.245171 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.245149 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.299764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.299740 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6"] Nov 03 00:47:31.343794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.343771 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:31.343794 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.343784 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:47:31.351392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.351371 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.351474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.351399 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbs54\" (UniqueName: \"kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.351516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.351498 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.452723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.452694 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.452859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.452726 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbs54\" (UniqueName: \"kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.452859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.452783 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.453359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.453340 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.455166 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.455142 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.509145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.509123 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbs54\" (UniqueName: \"kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54\") pod \"maas-api-revoke-sub-5a8ee86d-7zqb6\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.559468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.559444 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:31.936985 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:47:31.936950 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ce3d85e_1853_4e62_938b_ccd54485dd78.slice/crio-57d5260d3938afa531e7fc7430ed7c7000586a7101541dddb9ddad6e45af517f WatchSource:0}: Error finding container 57d5260d3938afa531e7fc7430ed7c7000586a7101541dddb9ddad6e45af517f: Status 404 returned error can't find the container with id 57d5260d3938afa531e7fc7430ed7c7000586a7101541dddb9ddad6e45af517f Nov 03 00:47:31.938816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:31.938792 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6"] Nov 03 00:47:32.046995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:32.046972 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt"] Nov 03 00:47:32.051051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:32.051030 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68eda06d-tbfgt"] Nov 03 00:47:32.214999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:32.214970 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" event={"ID":"8ce3d85e-1853-4e62-938b-ccd54485dd78","Type":"ContainerStarted","Data":"57d5260d3938afa531e7fc7430ed7c7000586a7101541dddb9ddad6e45af517f"} Nov 03 00:47:33.230149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:33.230061 5144 generic.go:356] "Generic (PLEG): container finished" podID="8ce3d85e-1853-4e62-938b-ccd54485dd78" containerID="f131552bfd0b081db6cbd93736c2a3944f19da420132882fe7e1502be2f92161" exitCode=0 Nov 03 00:47:33.230149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:33.230120 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" event={"ID":"8ce3d85e-1853-4e62-938b-ccd54485dd78","Type":"ContainerDied","Data":"f131552bfd0b081db6cbd93736c2a3944f19da420132882fe7e1502be2f92161"} Nov 03 00:47:33.998424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:33.998388 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80abe2c6-9de2-4dc7-96e0-ea92c5deee13" path="/var/lib/kubelet/pods/80abe2c6-9de2-4dc7-96e0-ea92c5deee13/volumes" Nov 03 00:47:34.395602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.395576 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:34.490285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.490261 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca\") pod \"8ce3d85e-1853-4e62-938b-ccd54485dd78\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " Nov 03 00:47:34.490473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.490321 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca\") pod \"8ce3d85e-1853-4e62-938b-ccd54485dd78\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " Nov 03 00:47:34.490473 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.490394 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54\" (UniqueName: \"kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54\") pod \"8ce3d85e-1853-4e62-938b-ccd54485dd78\" (UID: \"8ce3d85e-1853-4e62-938b-ccd54485dd78\") " Nov 03 00:47:34.490698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.490675 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca" pod "8ce3d85e-1853-4e62-938b-ccd54485dd78" (UID: "8ce3d85e-1853-4e62-938b-ccd54485dd78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:34.490829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.490811 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:34.492712 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.492689 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54" pod "8ce3d85e-1853-4e62-938b-ccd54485dd78" (UID: "8ce3d85e-1853-4e62-938b-ccd54485dd78"). InnerVolumeSpecName "kube-api-access-hbs54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:34.492859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.492841 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca" pod "8ce3d85e-1853-4e62-938b-ccd54485dd78" (UID: "8ce3d85e-1853-4e62-938b-ccd54485dd78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:34.591779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.591689 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbs54\" (UniqueName: \"kubernetes.io/projected/8ce3d85e-1853-4e62-938b-ccd54485dd78-kube-api-access-hbs54\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:34.591779 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:34.591727 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ce3d85e-1853-4e62-938b-ccd54485dd78-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:35.251350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:35.251320 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" Nov 03 00:47:35.251540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:35.251316 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6" event={"ID":"8ce3d85e-1853-4e62-938b-ccd54485dd78","Type":"ContainerDied","Data":"57d5260d3938afa531e7fc7430ed7c7000586a7101541dddb9ddad6e45af517f"} Nov 03 00:47:35.251540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:35.251458 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57d5260d3938afa531e7fc7430ed7c7000586a7101541dddb9ddad6e45af517f" Nov 03 00:47:41.946826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:41.946788 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd"] Nov 03 00:47:41.947770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:41.947746 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8ce3d85e-1853-4e62-938b-ccd54485dd78" containerName="revoke-keys" Nov 03 00:47:41.947770 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:41.947765 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ce3d85e-1853-4e62-938b-ccd54485dd78" containerName="revoke-keys" Nov 03 00:47:41.947945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:41.947894 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ce3d85e-1853-4e62-938b-ccd54485dd78" containerName="revoke-keys" Nov 03 00:47:41.952878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:41.952856 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.001120 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.001093 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd"] Nov 03 00:47:42.045273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.045244 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:47:42.045458 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.045300 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:47:42.101261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.101227 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.101428 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.101282 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.101504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.101428 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w86m6\" (UniqueName: \"kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.164167 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:47:42.164130 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:47:42.202481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.202419 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.202481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.202458 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.202691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.202492 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w86m6\" (UniqueName: \"kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.203161 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.203139 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.204809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.204791 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.259191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.259160 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w86m6\" (UniqueName: \"kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6\") pod \"maas-api-revoke-sub-d27b0c3e-qncpd\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.267049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.267024 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:42.667768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:42.667740 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd"] Nov 03 00:47:42.668518 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:47:42.668495 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee224846_7e83_4742_ac0d_d923e316391e.slice/crio-103db1fcd83decf090a07d2ef050c28c507295d230cc8a8d734a497909cb0461 WatchSource:0}: Error finding container 103db1fcd83decf090a07d2ef050c28c507295d230cc8a8d734a497909cb0461: Status 404 returned error can't find the container with id 103db1fcd83decf090a07d2ef050c28c507295d230cc8a8d734a497909cb0461 Nov 03 00:47:43.350734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:43.350640 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" event={"ID":"ee224846-7e83-4742-ac0d-d923e316391e","Type":"ContainerStarted","Data":"b14b84c60438be0154a0380f19fa2c0cdab4dae9f4ad2b33c4c1b9cfefb6ea75"} Nov 03 00:47:43.350734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:43.350692 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" event={"ID":"ee224846-7e83-4742-ac0d-d923e316391e","Type":"ContainerStarted","Data":"103db1fcd83decf090a07d2ef050c28c507295d230cc8a8d734a497909cb0461"} Nov 03 00:47:43.466519 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:43.466464 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" podStartSLOduration=2.466448722 podStartE2EDuration="2.466448722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:43.463372349 +0000 UTC m=+2553.609066818" watchObservedRunningTime="2026-11-03 00:47:43.466448722 +0000 UTC m=+2553.612143178" Nov 03 00:47:44.361688 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:44.361649 5144 generic.go:356] "Generic (PLEG): container finished" podID="ee224846-7e83-4742-ac0d-d923e316391e" containerID="b14b84c60438be0154a0380f19fa2c0cdab4dae9f4ad2b33c4c1b9cfefb6ea75" exitCode=0 Nov 03 00:47:44.362115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:44.361745 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" event={"ID":"ee224846-7e83-4742-ac0d-d923e316391e","Type":"ContainerDied","Data":"b14b84c60438be0154a0380f19fa2c0cdab4dae9f4ad2b33c4c1b9cfefb6ea75"} Nov 03 00:47:45.541923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.541400 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:45.673579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.673504 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6\" (UniqueName: \"kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6\") pod \"ee224846-7e83-4742-ac0d-d923e316391e\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " Nov 03 00:47:45.673782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.673605 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca\") pod \"ee224846-7e83-4742-ac0d-d923e316391e\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " Nov 03 00:47:45.673782 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.673732 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca\") pod \"ee224846-7e83-4742-ac0d-d923e316391e\" (UID: \"ee224846-7e83-4742-ac0d-d923e316391e\") " Nov 03 00:47:45.674141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.674089 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca" pod "ee224846-7e83-4742-ac0d-d923e316391e" (UID: "ee224846-7e83-4742-ac0d-d923e316391e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:47:45.674424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.674403 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:45.676104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.676077 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6" pod "ee224846-7e83-4742-ac0d-d923e316391e" (UID: "ee224846-7e83-4742-ac0d-d923e316391e"). InnerVolumeSpecName "kube-api-access-w86m6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:47:45.676390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.676370 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca" pod "ee224846-7e83-4742-ac0d-d923e316391e" (UID: "ee224846-7e83-4742-ac0d-d923e316391e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:47:45.776147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.776109 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w86m6\" (UniqueName: \"kubernetes.io/projected/ee224846-7e83-4742-ac0d-d923e316391e-kube-api-access-w86m6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:45.776147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:45.776146 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee224846-7e83-4742-ac0d-d923e316391e-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:47:46.381235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:46.381150 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" Nov 03 00:47:46.381235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:46.381188 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd" event={"ID":"ee224846-7e83-4742-ac0d-d923e316391e","Type":"ContainerDied","Data":"103db1fcd83decf090a07d2ef050c28c507295d230cc8a8d734a497909cb0461"} Nov 03 00:47:46.381426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:46.381245 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="103db1fcd83decf090a07d2ef050c28c507295d230cc8a8d734a497909cb0461" Nov 03 00:47:48.054997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.054963 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj"] Nov 03 00:47:48.103732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.103705 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-daab9573-mwhqj"] Nov 03 00:47:48.400284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.400187 5144 generic.go:356] "Generic (PLEG): container finished" podID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerID="a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03" exitCode=0 Nov 03 00:47:48.400284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.400262 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" event={"ID":"6052680c-30b2-44f4-ac41-f24f33dc8fca","Type":"ContainerDied","Data":"a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03"} Nov 03 00:47:48.400743 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.400728 5144 scope.go:122] "RemoveContainer" containerID="a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03" Nov 03 00:47:48.402434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.402409 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:47:48.402504 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.402476 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:47:48.402553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:48.402506 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:47:49.411402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:49.411329 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" event={"ID":"6052680c-30b2-44f4-ac41-f24f33dc8fca","Type":"ContainerStarted","Data":"07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe"} Nov 03 00:47:49.830656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:49.829006 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:47:49.830869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:49.830731 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 00:47:49.830869 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:49.830791 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 00:47:49.997278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:49.997247 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f0f439-2e96-4093-acdd-1d57446f4b63" path="/var/lib/kubelet/pods/82f0f439-2e96-4093-acdd-1d57446f4b63/volumes" Nov 03 00:47:50.834973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:50.834935 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 00:47:50.835578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:50.835543 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 00:47:51.826658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:51.823868 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 00:47:51.826658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:51.823927 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 00:47:52.831657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:52.828918 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 00:47:52.831657 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:52.828975 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 00:47:53.829658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:53.825931 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" start-of-body= Nov 03 00:47:53.829658 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:53.825991 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.238:15021/healthz/ready\": dial tcp 10.132.0.238:15021: connect: connection refused" Nov 03 00:47:54.828202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.828173 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:47:54.828683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.828570 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:47:54.829321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.829301 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 00:47:54.985130 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.985094 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t"] Nov 03 00:47:54.985806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.985791 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ee224846-7e83-4742-ac0d-d923e316391e" containerName="revoke-keys" Nov 03 00:47:54.985872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.985807 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee224846-7e83-4742-ac0d-d923e316391e" containerName="revoke-keys" Nov 03 00:47:54.985924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.985915 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee224846-7e83-4742-ac0d-d923e316391e" containerName="revoke-keys" Nov 03 00:47:54.990026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:54.990006 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.016798 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.016870 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.016955 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.017398 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.017521 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.017610 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.017873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.017686 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbg7v\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.019254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.019201 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.019572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.019549 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.126955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.126472 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.126955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.126537 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.126955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.126786 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.127540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.127352 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.127540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.127441 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.127540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.127495 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.127817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.127562 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.127817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.127655 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.127817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.127695 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbg7v\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.128564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.128025 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.128564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.128134 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.128564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.128302 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.128564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.128389 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.128910 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.128561 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.130699 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.130677 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-c39710-openshift-default-dockercfg-vtbfh\"" Nov 03 00:47:55.162407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.162359 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.183555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.183460 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t"] Nov 03 00:47:55.189013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.188975 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.190533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.190485 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbg7v\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.212695 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.212663 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy\") pod \"e2e-explicit-c39710-openshift-default-765c567d77-szb8t\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.304918 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.304888 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:55.756338 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:47:55.756295 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cfbf58_d149_459b_a6cc_ca200c99029f.slice/crio-fbd940cdfd1ab670733198eb8cadc9ef5d19b954397bed50e80f067846f044b8 WatchSource:0}: Error finding container fbd940cdfd1ab670733198eb8cadc9ef5d19b954397bed50e80f067846f044b8: Status 404 returned error can't find the container with id fbd940cdfd1ab670733198eb8cadc9ef5d19b954397bed50e80f067846f044b8 Nov 03 00:47:55.756588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.756566 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t"] Nov 03 00:47:55.759295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.759255 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:47:55.759386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.759338 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:47:55.759386 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:55.759379 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:47:56.492916 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:56.492880 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" event={"ID":"35cfbf58-d149-459b-a6cc-ca200c99029f","Type":"ContainerStarted","Data":"97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028"} Nov 03 00:47:56.493326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:56.492935 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" event={"ID":"35cfbf58-d149-459b-a6cc-ca200c99029f","Type":"ContainerStarted","Data":"fbd940cdfd1ab670733198eb8cadc9ef5d19b954397bed50e80f067846f044b8"} Nov 03 00:47:56.664147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:56.664092 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podStartSLOduration=2.664078894 podStartE2EDuration="2.664078894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:56.660049314 +0000 UTC m=+2566.805743771" watchObservedRunningTime="2026-11-03 00:47:56.664078894 +0000 UTC m=+2566.809773348" Nov 03 00:47:57.305026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:57.304993 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:57.310833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:57.310800 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:57.500689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:57.500659 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:57.501711 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:57.501691 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:47:58.048381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:58.048350 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q"] Nov 03 00:47:58.052563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:58.052540 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8e1f58-jwl2q"] Nov 03 00:47:59.997407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:47:59.997363 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a81826-7db1-4b9a-9b2e-3132d4bf9fe1" path="/var/lib/kubelet/pods/65a81826-7db1-4b9a-9b2e-3132d4bf9fe1/volumes" Nov 03 00:48:03.355135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.355092 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs"] Nov 03 00:48:03.385074 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.385036 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.392987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.391377 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs"] Nov 03 00:48:03.451463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.451433 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.451692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.451490 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8k42\" (UniqueName: \"kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42\") pod \"payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.487128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.487091 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-c39710\"" Nov 03 00:48:03.487128 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.487103 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-c39710-dockercfg-nh4pn\"" Nov 03 00:48:03.510337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.510312 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5"] Nov 03 00:48:03.514826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.514805 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.553593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.553548 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.553741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.553615 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.553741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.553724 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8k42\" (UniqueName: \"kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42\") pod \"payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.553935 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.553912 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txb46\" (UniqueName: \"kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46\") pod \"payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.555792 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.555767 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume\") pod \"payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.572205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.572175 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5"] Nov 03 00:48:03.612710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.612651 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8k42\" (UniqueName: \"kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42\") pod \"payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.655786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.655763 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txb46\" (UniqueName: \"kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46\") pod \"payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.655926 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.655836 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.656466 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.656448 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.714877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.714853 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txb46\" (UniqueName: \"kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46\") pod \"payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:03.732758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.732736 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:03.829523 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:03.829277 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:04.049534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.049502 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b"] Nov 03 00:48:04.056243 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.056216 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-095dd76a-rhz8b"] Nov 03 00:48:04.157007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.156976 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs"] Nov 03 00:48:04.157529 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:48:04.157501 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41df7b0e_61a2_4d06_8557_b75e9e7ac0a5.slice/crio-5af2b8e1c101ee5d7d2a4bdfe768ced6a11a7f2a33c36a0982a3027f303426d0 WatchSource:0}: Error finding container 5af2b8e1c101ee5d7d2a4bdfe768ced6a11a7f2a33c36a0982a3027f303426d0: Status 404 returned error can't find the container with id 5af2b8e1c101ee5d7d2a4bdfe768ced6a11a7f2a33c36a0982a3027f303426d0 Nov 03 00:48:04.230882 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.230851 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5"] Nov 03 00:48:04.231012 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:48:04.230894 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb67e56_66d4_42aa_9bc2_d820c2a85e96.slice/crio-7661c733b7b4888ba6ee65312a2b2ab79f9130d74873174e02b6342d9135b5e1 WatchSource:0}: Error finding container 7661c733b7b4888ba6ee65312a2b2ab79f9130d74873174e02b6342d9135b5e1: Status 404 returned error can't find the container with id 7661c733b7b4888ba6ee65312a2b2ab79f9130d74873174e02b6342d9135b5e1 Nov 03 00:48:04.561979 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.561943 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" event={"ID":"dbb67e56-66d4-42aa-9bc2-d820c2a85e96","Type":"ContainerStarted","Data":"141a438131395705e2f1e6c617c61c2c6e8f13ff48633c74fdf34ca57cb63afb"} Nov 03 00:48:04.562475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.562007 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" event={"ID":"dbb67e56-66d4-42aa-9bc2-d820c2a85e96","Type":"ContainerStarted","Data":"7661c733b7b4888ba6ee65312a2b2ab79f9130d74873174e02b6342d9135b5e1"} Nov 03 00:48:04.562475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.562041 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:04.564083 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.564056 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" event={"ID":"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5","Type":"ContainerStarted","Data":"899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c"} Nov 03 00:48:04.564208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.564113 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:04.564208 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.564136 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" event={"ID":"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5","Type":"ContainerStarted","Data":"5af2b8e1c101ee5d7d2a4bdfe768ced6a11a7f2a33c36a0982a3027f303426d0"} Nov 03 00:48:04.675742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.675700 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" podStartSLOduration=1.675686749 podStartE2EDuration="1.675686749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:04.673465436 +0000 UTC m=+2574.819159909" watchObservedRunningTime="2026-11-03 00:48:04.675686749 +0000 UTC m=+2574.821381205" Nov 03 00:48:04.808265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:04.808215 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" podStartSLOduration=1.808200966 podStartE2EDuration="1.808200966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:04.803187055 +0000 UTC m=+2574.948881530" watchObservedRunningTime="2026-11-03 00:48:04.808200966 +0000 UTC m=+2574.953895421" Nov 03 00:48:05.998435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:05.998401 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec55f256-9d9d-4232-a41e-f61e0aea18df" path="/var/lib/kubelet/pods/ec55f256-9d9d-4232-a41e-f61e0aea18df/volumes" Nov 03 00:48:12.164140 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:48:12.164093 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:48:12.239366 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:12.239330 5144 scope.go:122] "RemoveContainer" containerID="3302488a0a62d1cb10a4b28a1d390e8849f498f3dfb6a81634476202bafc6894" Nov 03 00:48:14.050059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:14.050020 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx"] Nov 03 00:48:14.099194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:14.099156 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-047a1eda-n69sx"] Nov 03 00:48:15.048802 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:15.048758 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv"] Nov 03 00:48:15.098479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:15.098450 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7c591fb-d5bkv"] Nov 03 00:48:15.575424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:15.575399 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:48:15.575602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:15.575486 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:48:15.999155 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:15.999128 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8791297c-2e96-48b9-97df-276f4089a227" path="/var/lib/kubelet/pods/8791297c-2e96-48b9-97df-276f4089a227/volumes" Nov 03 00:48:15.999578 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:15.999564 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e7112cc-f328-43e6-9436-4a0f84a8e055" path="/var/lib/kubelet/pods/8e7112cc-f328-43e6-9436-4a0f84a8e055/volumes" Nov 03 00:48:19.712928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:19.712834 5144 generic.go:356] "Generic (PLEG): container finished" podID="f46ed093-2590-4248-ad19-409dd68099a1" containerID="75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93" exitCode=0 Nov 03 00:48:19.713431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:19.712930 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" event={"ID":"f46ed093-2590-4248-ad19-409dd68099a1","Type":"ContainerDied","Data":"75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93"} Nov 03 00:48:19.713431 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:19.713396 5144 scope.go:122] "RemoveContainer" containerID="75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93" Nov 03 00:48:19.715401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:19.715368 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:48:19.715494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:19.715460 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:48:19.715552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:19.715507 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:48:20.747380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:20.747344 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" event={"ID":"f46ed093-2590-4248-ad19-409dd68099a1","Type":"ContainerStarted","Data":"042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f"} Nov 03 00:48:21.038656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:21.037981 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:48:21.043049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:21.043017 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 00:48:21.043219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:21.043083 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 00:48:21.687818 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:21.687786 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:48:22.038971 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:22.038936 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 00:48:22.039444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:22.038994 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 00:48:23.040492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:23.040017 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 00:48:23.040492 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:23.040073 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 00:48:24.037990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:24.037948 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 00:48:24.038288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:24.038259 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 00:48:25.039028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:25.038990 5144 patch_prober.go:29] interesting pod/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" start-of-body= Nov 03 00:48:25.039621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:25.039587 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.239:15021/healthz/ready\": dial tcp 10.132.0.239:15021: connect: connection refused" Nov 03 00:48:26.042484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:26.042450 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:48:26.043599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:26.043576 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 00:48:38.046668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:38.046616 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469"] Nov 03 00:48:38.099707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:38.099667 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-667efdf8-4j469"] Nov 03 00:48:39.998964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:39.998929 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8bb83b0-2b44-4559-b9ab-97924f9c8f0f" path="/var/lib/kubelet/pods/b8bb83b0-2b44-4559-b9ab-97924f9c8f0f/volumes" Nov 03 00:48:42.160023 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:48:42.159985 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:48:44.045999 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:44.045962 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d"] Nov 03 00:48:44.073165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:44.073129 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022f7cf2-kv92d"] Nov 03 00:48:45.048680 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:45.048620 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb"] Nov 03 00:48:45.095565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:45.095534 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cd2322f-m7fdb"] Nov 03 00:48:45.997994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:45.997960 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25698ac1-1c6b-40d4-8cdc-349692af65be" path="/var/lib/kubelet/pods/25698ac1-1c6b-40d4-8cdc-349692af65be/volumes" Nov 03 00:48:45.998446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:48:45.998430 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2ed77bd-0674-4f6f-92d7-d61592b2bbb5" path="/var/lib/kubelet/pods/b2ed77bd-0674-4f6f-92d7-d61592b2bbb5/volumes" Nov 03 00:49:00.047857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:00.047822 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6"] Nov 03 00:49:00.097104 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:00.097074 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-faaece96-jrxp6"] Nov 03 00:49:00.100959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:00.100935 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp"] Nov 03 00:49:00.104415 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:00.104393 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5131c990-kwlhp"] Nov 03 00:49:02.002206 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:02.002167 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1a2756-8359-431d-8c11-c00d8504fdd1" path="/var/lib/kubelet/pods/0a1a2756-8359-431d-8c11-c00d8504fdd1/volumes" Nov 03 00:49:02.003438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:02.003415 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a968356-24f7-45de-9916-bc4d2769d1c6" path="/var/lib/kubelet/pods/1a968356-24f7-45de-9916-bc4d2769d1c6/volumes" Nov 03 00:49:10.048582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:10.048544 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4"] Nov 03 00:49:10.097725 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:10.097693 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c120a8d-w78f4"] Nov 03 00:49:11.997500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:11.997467 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5745d317-84cc-4bf2-9a4d-1017245c2cbf" path="/var/lib/kubelet/pods/5745d317-84cc-4bf2-9a4d-1017245c2cbf/volumes" Nov 03 00:49:12.154879 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:49:12.154844 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:49:12.274516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.274483 5144 scope.go:122] "RemoveContainer" containerID="7980b897ce2a35a883658421102e85b6a993251670ec5aac25263e6778db338c" Nov 03 00:49:12.285476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.285459 5144 scope.go:122] "RemoveContainer" containerID="77392348e73fe7e08ab6fff46a146b6c118c8f9f93ee39e60041fbdb91af0685" Nov 03 00:49:12.295144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.295123 5144 scope.go:122] "RemoveContainer" containerID="8ec844b4d93ec10ea850cd7a25cf541f46894bf8b60787f3e5e50e20a744cde4" Nov 03 00:49:12.305443 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.305425 5144 scope.go:122] "RemoveContainer" containerID="50666c35c51940694bffb2de11ab89fe1ca6f82d00544db7f014a7b705e88d5e" Nov 03 00:49:12.315187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.315170 5144 scope.go:122] "RemoveContainer" containerID="e28bf4287570869977f7f112e8d21294938ea82b09f0ee560098aacfd5a05fe8" Nov 03 00:49:12.325003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.324981 5144 scope.go:122] "RemoveContainer" containerID="85d37b9e44fff045aa4e956fb1e311a967bf632cd44ee9b8c78ad0a9abec64a3" Nov 03 00:49:12.336020 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:12.336001 5144 scope.go:122] "RemoveContainer" containerID="9b01f62b09f56c99d8e19263fa97fea9e2d0684f63b422246fa17195b326de72" Nov 03 00:49:21.047927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:21.047844 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8"] Nov 03 00:49:21.096707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:21.096678 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3d56fe-zmpj8"] Nov 03 00:49:22.000889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:22.000854 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146b5fbd-12ff-49ca-b6d1-81d24e85ec8a" path="/var/lib/kubelet/pods/146b5fbd-12ff-49ca-b6d1-81d24e85ec8a/volumes" Nov 03 00:49:23.046790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:23.046760 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d"] Nov 03 00:49:23.103017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:23.102980 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43342ee6-d628d"] Nov 03 00:49:23.997317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:23.997287 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f8da45-cd7b-4e11-9d35-70559dcae73b" path="/var/lib/kubelet/pods/47f8da45-cd7b-4e11-9d35-70559dcae73b/volumes" Nov 03 00:49:32.423362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:32.423264 5144 generic.go:356] "Generic (PLEG): container finished" podID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerID="97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028" exitCode=0 Nov 03 00:49:32.423362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:32.423334 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" event={"ID":"35cfbf58-d149-459b-a6cc-ca200c99029f","Type":"ContainerDied","Data":"97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028"} Nov 03 00:49:32.423984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:32.423815 5144 scope.go:122] "RemoveContainer" containerID="97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028" Nov 03 00:49:32.425775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:32.425741 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:49:32.425913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:32.425831 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:49:32.425913 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:32.425877 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:49:33.046789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:33.046755 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46"] Nov 03 00:49:33.052055 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:33.052027 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c5035b-zzg46"] Nov 03 00:49:33.458296 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:33.458209 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" event={"ID":"35cfbf58-d149-459b-a6cc-ca200c99029f","Type":"ContainerStarted","Data":"2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9"} Nov 03 00:49:33.997019 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:33.996986 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1742329-5a3d-455d-a189-8c1ddcaee582" path="/var/lib/kubelet/pods/f1742329-5a3d-455d-a189-8c1ddcaee582/volumes" Nov 03 00:49:34.306039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:34.306009 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:49:34.306269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:34.306249 5144 patch_prober.go:29] interesting pod/e2e-explicit-c39710-openshift-default-765c567d77-szb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 00:49:34.306376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:34.306300 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 00:49:35.306126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:35.306093 5144 patch_prober.go:29] interesting pod/e2e-explicit-c39710-openshift-default-765c567d77-szb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 00:49:35.306547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:35.306157 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 00:49:36.097732 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:36.097695 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc"] Nov 03 00:49:36.147686 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:36.147649 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41790f4e-vbmjc"] Nov 03 00:49:36.305188 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:36.305155 5144 patch_prober.go:29] interesting pod/e2e-explicit-c39710-openshift-default-765c567d77-szb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 00:49:36.305376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:36.305216 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 00:49:37.305354 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:37.305322 5144 patch_prober.go:29] interesting pod/e2e-explicit-c39710-openshift-default-765c567d77-szb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 00:49:37.305804 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:37.305419 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 00:49:37.997728 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:37.997692 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3832aeb6-32ce-4bc7-8e55-1a4070abf13f" path="/var/lib/kubelet/pods/3832aeb6-32ce-4bc7-8e55-1a4070abf13f/volumes" Nov 03 00:49:38.305359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:38.305328 5144 patch_prober.go:29] interesting pod/e2e-explicit-c39710-openshift-default-765c567d77-szb8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body= Nov 03 00:49:38.305775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:38.305390 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" Nov 03 00:49:39.045870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:39.045815 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747"] Nov 03 00:49:39.097430 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:39.097401 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d2a92a0-fd747"] Nov 03 00:49:39.309962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:39.309885 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:49:39.310424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:39.310270 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:49:39.310905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:39.310886 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:49:39.998806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:39.998777 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="143ed4f6-71c3-41e7-9a4d-7e1b7dd17489" path="/var/lib/kubelet/pods/143ed4f6-71c3-41e7-9a4d-7e1b7dd17489/volumes" Nov 03 00:49:42.159775 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:49:42.159739 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:49:46.055367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:46.055312 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg"] Nov 03 00:49:46.098842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:46.098817 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9fd792be-7hpfg"] Nov 03 00:49:47.998105 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:47.998062 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55f90be2-5fce-4eaf-8bf2-4faf409c2a33" path="/var/lib/kubelet/pods/55f90be2-5fce-4eaf-8bf2-4faf409c2a33/volumes" Nov 03 00:49:50.442850 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:50.442806 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d"] Nov 03 00:49:50.443359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:50.443049 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" containerID="cri-o://7f495125f2af637428c61eead0d3a780ac427b6238916e981fedfad71e0dc22b" gracePeriod=30 Nov 03 00:49:55.645353 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.645318 5144 generic.go:356] "Generic (PLEG): container finished" podID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerID="7f495125f2af637428c61eead0d3a780ac427b6238916e981fedfad71e0dc22b" exitCode=0 Nov 03 00:49:55.645838 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.645549 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" event={"ID":"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e","Type":"ContainerDied","Data":"7f495125f2af637428c61eead0d3a780ac427b6238916e981fedfad71e0dc22b"} Nov 03 00:49:55.723210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.723155 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:49:55.789565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789535 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.789733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789572 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.789733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789626 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.789733 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789678 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.789873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789745 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.789922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789870 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.789922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789902 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.790010 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.789994 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.790073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790029 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.790073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790036 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.790073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790049 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.790239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790085 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token\") pod \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\" (UID: \"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e\") " Nov 03 00:49:55.790239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790138 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.790468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790441 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:49:55.790554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790457 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.790554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790494 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.790554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790510 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.790554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.790524 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.791978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.791947 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:49:55.792288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.792268 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:49:55.792392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.792372 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:49:55.792432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.792406 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh" pod "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" (UID: "3774f1b2-87c9-4fb4-b67a-7cc7c808a60e"). InnerVolumeSpecName "kube-api-access-tdddh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:49:55.891866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.891837 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdddh\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-kube-api-access-tdddh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.891866 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.891868 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.892064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.891886 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.892064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.891899 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:55.892064 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:55.891908 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:49:56.654140 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:56.654110 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" Nov 03 00:49:56.654541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:56.654140 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d" event={"ID":"3774f1b2-87c9-4fb4-b67a-7cc7c808a60e","Type":"ContainerDied","Data":"052294b6f42e07fbf1756b02e8b9491d48de11a2da527f459de2570fefb111f8"} Nov 03 00:49:56.654541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:56.654221 5144 scope.go:122] "RemoveContainer" containerID="7f495125f2af637428c61eead0d3a780ac427b6238916e981fedfad71e0dc22b" Nov 03 00:49:56.869127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:56.869089 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d"] Nov 03 00:49:56.917829 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:56.917773 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-4sc5d"] Nov 03 00:49:57.996942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:49:57.996911 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" path="/var/lib/kubelet/pods/3774f1b2-87c9-4fb4-b67a-7cc7c808a60e/volumes" Nov 03 00:50:00.460078 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.460046 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq"] Nov 03 00:50:00.461107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.461086 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" Nov 03 00:50:00.461225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.461109 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" Nov 03 00:50:00.461281 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.461254 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3774f1b2-87c9-4fb4-b67a-7cc7c808a60e" containerName="istio-proxy" Nov 03 00:50:00.465402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.465383 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.520384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.520355 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq"] Nov 03 00:50:00.548903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.548870 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whsb5\" (UniqueName: \"kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.549038 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.548943 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.549181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.549155 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.549256 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.549240 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.549293 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.549278 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.549350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.549336 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.563945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.563909 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4c8b1\"/\"kube-root-ca.crt\"" Nov 03 00:50:00.563945 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.563916 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4c8b1\"/\"openshift-service-ca.crt\"" Nov 03 00:50:00.564136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.563963 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4c8b1\"/\"default-dockercfg-4sjxh\"" Nov 03 00:50:00.566671 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.566652 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-e4c8b1\"/\"auth-test-model-e4c8b1-kserve-self-signed-certs\"" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.652894 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.653001 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.653042 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.653108 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.653151 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whsb5\" (UniqueName: \"kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.653383 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.655031 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.655287 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.656396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.655516 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.657207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.656701 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.666755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.666725 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.715138 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.715076 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whsb5\" (UniqueName: \"kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5\") pod \"auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:00.778401 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:00.778210 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:01.161616 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:50:01.161586 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod076dff53_bf61_4a5b_8789_8d6342df3a70.slice/crio-ec2c1e4747e898d7266a6e6357a30fba1b9b307ab88ba14e2721529e95f8e4ed WatchSource:0}: Error finding container ec2c1e4747e898d7266a6e6357a30fba1b9b307ab88ba14e2721529e95f8e4ed: Status 404 returned error can't find the container with id ec2c1e4747e898d7266a6e6357a30fba1b9b307ab88ba14e2721529e95f8e4ed Nov 03 00:50:01.163214 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:01.163187 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq"] Nov 03 00:50:01.698696 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:01.698659 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" event={"ID":"076dff53-bf61-4a5b-8789-8d6342df3a70","Type":"ContainerStarted","Data":"758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5"} Nov 03 00:50:01.699094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:01.698713 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" event={"ID":"076dff53-bf61-4a5b-8789-8d6342df3a70","Type":"ContainerStarted","Data":"ec2c1e4747e898d7266a6e6357a30fba1b9b307ab88ba14e2721529e95f8e4ed"} Nov 03 00:50:01.778932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:01.778902 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:01.813088 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:01.813046 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" podStartSLOduration=1.813032381 podStartE2EDuration="1.813032381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:01.811304273 +0000 UTC m=+2691.956998740" watchObservedRunningTime="2026-11-03 00:50:01.813032381 +0000 UTC m=+2691.958726836" Nov 03 00:50:03.050358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:03.050316 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw"] Nov 03 00:50:03.097615 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:03.097591 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-487dea6d-tv6qw"] Nov 03 00:50:03.997571 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:03.997536 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c396136-969a-4515-bfc1-e2dd538eb5bc" path="/var/lib/kubelet/pods/6c396136-969a-4515-bfc1-e2dd538eb5bc/volumes" Nov 03 00:50:10.623757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:10.623728 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:50:10.632455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:10.632420 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:50:10.779312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:10.779281 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:10.791581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:10.791557 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:11.788775 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:11.788749 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:50:12.155214 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:50:12.155122 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:50:12.401690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:12.401662 5144 scope.go:122] "RemoveContainer" containerID="7d3b17bd27a9fac25ede8b5a06a62770b5e51e7685cb1417c7c296594eee7c84" Nov 03 00:50:12.412255 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:12.412228 5144 scope.go:122] "RemoveContainer" containerID="d89870d1b2fd6c57df18a5781f000c56719eeb86979d45babfa00f5b55b849bf" Nov 03 00:50:12.421542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:12.421522 5144 scope.go:122] "RemoveContainer" containerID="5ab382cbd80a45717d65e4b19153843bae5d5baebd1ac2c6020332787ec8c6e6" Nov 03 00:50:12.431107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:12.431090 5144 scope.go:122] "RemoveContainer" containerID="84a0e71f371816736b533de9fee2b234be9b579b5c8d65693445b827ab0e83ee" Nov 03 00:50:12.443497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:12.440471 5144 scope.go:122] "RemoveContainer" containerID="98c85584fad5f7ed45e6a4fbdf7710e8455ad1705926397f42004f91f40df434" Nov 03 00:50:12.452434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:12.452414 5144 scope.go:122] "RemoveContainer" containerID="2a4c68ef86da8938d18e189559fe799f655ff9a114bcf82607056d9682f5c2f9" Nov 03 00:50:39.054751 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:39.054713 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc"] Nov 03 00:50:39.101961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:39.101935 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-59cda756-p87tc"] Nov 03 00:50:40.006805 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:40.006771 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c022bb8d-cb70-4f60-9f68-b28b43e080c3" path="/var/lib/kubelet/pods/c022bb8d-cb70-4f60-9f68-b28b43e080c3/volumes" Nov 03 00:50:42.155776 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:50:42.155732 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:50:43.753102 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:43.753071 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr"] Nov 03 00:50:43.753599 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:43.753300 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" containerID="cri-o://eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84" gracePeriod=30 Nov 03 00:50:44.027758 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.027687 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9"] Nov 03 00:50:44.032258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.032232 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.060107 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.060082 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.060283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.060134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.060283 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.060253 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.060440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.060287 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.060670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.060620 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w4sv\" (UniqueName: \"kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.060805 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.060742 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.087048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.087024 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"kube-root-ca.crt\"" Nov 03 00:50:44.087190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.087104 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"openshift-service-ca.crt\"" Nov 03 00:50:44.087190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.087159 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"auth-test-model-fce94b-kserve-self-signed-certs\"" Nov 03 00:50:44.087190 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.087170 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-fce94b\"/\"default-dockercfg-4mz2z\"" Nov 03 00:50:44.090567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.090544 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9"] Nov 03 00:50:44.161529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161502 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.161702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161547 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.161702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161616 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w4sv\" (UniqueName: \"kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.161836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161731 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.161836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161828 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.161957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161882 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.162306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.161968 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.162467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.162388 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.162587 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.162535 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.189326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.189284 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.204202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.204165 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.218452 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.218426 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w4sv\" (UniqueName: \"kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv\") pod \"auth-test-model-fce94b-kserve-6fd858f87c-tjcx9\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.345789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.345704 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:44.724247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:44.724213 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9"] Nov 03 00:50:44.724435 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:50:44.724403 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c79a872_0552_41d7_8bb2_686de23704aa.slice/crio-84097f8a2886233f734971514589536088ce91a68379fc759fd3b9ea8e26e334 WatchSource:0}: Error finding container 84097f8a2886233f734971514589536088ce91a68379fc759fd3b9ea8e26e334: Status 404 returned error can't find the container with id 84097f8a2886233f734971514589536088ce91a68379fc759fd3b9ea8e26e334 Nov 03 00:50:45.064081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:45.064049 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" event={"ID":"5c79a872-0552-41d7-8bb2-686de23704aa","Type":"ContainerStarted","Data":"2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95"} Nov 03 00:50:45.064565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:45.064090 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" event={"ID":"5c79a872-0552-41d7-8bb2-686de23704aa","Type":"ContainerStarted","Data":"84097f8a2886233f734971514589536088ce91a68379fc759fd3b9ea8e26e334"} Nov 03 00:50:45.183059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:45.183013 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" podStartSLOduration=2.182999425 podStartE2EDuration="2.182999425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:45.178170832 +0000 UTC m=+2735.323865305" watchObservedRunningTime="2026-11-03 00:50:45.182999425 +0000 UTC m=+2735.328693882" Nov 03 00:50:45.346565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:45.346496 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:49.025363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.025301 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:50:49.094930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.094864 5144 generic.go:356] "Generic (PLEG): container finished" podID="70eced92-3511-4c69-9b83-d599f3e0f449" containerID="eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84" exitCode=0 Nov 03 00:50:49.094930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.094915 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" Nov 03 00:50:49.095150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.094926 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" event={"ID":"70eced92-3511-4c69-9b83-d599f3e0f449","Type":"ContainerDied","Data":"eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84"} Nov 03 00:50:49.095150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.094989 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr" event={"ID":"70eced92-3511-4c69-9b83-d599f3e0f449","Type":"ContainerDied","Data":"7abd0354be24df5978f82ac368e35ef3469f093120eb492afb897b781b0a4092"} Nov 03 00:50:49.095150 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.095012 5144 scope.go:122] "RemoveContainer" containerID="eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84" Nov 03 00:50:49.109739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.107841 5144 scope.go:122] "RemoveContainer" containerID="eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84" Nov 03 00:50:49.109739 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:50:49.108211 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84\": container with ID starting with eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84 not found: ID does not exist" containerID="eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84" Nov 03 00:50:49.109739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.108244 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84"} err="failed to get container status \"eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84\": rpc error: code = NotFound desc = could not find container \"eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84\": container with ID starting with eb74354aebf1e3210032d6e15c2677ee1757167255865660aafab8f0d28b3b84 not found: ID does not exist" Nov 03 00:50:49.117689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.117667 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.117791 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.117778 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.117867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.117848 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.117904 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:49.118007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.117954 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118016 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118136 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118102 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118143 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118205 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118199 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:50:49.118358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118330 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs\") pod \"70eced92-3511-4c69-9b83-d599f3e0f449\" (UID: \"70eced92-3511-4c69-9b83-d599f3e0f449\") " Nov 03 00:50:49.118461 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118422 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:49.119436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118891 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.119436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118915 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70eced92-3511-4c69-9b83-d599f3e0f449-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.119436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118930 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.119436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.118890 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:49.119436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.119124 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:49.120435 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.120411 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "kube-api-access-fkglr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:49.120784 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.120763 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:50:49.120887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.120802 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:50:49.120995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.120978 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo" pod "70eced92-3511-4c69-9b83-d599f3e0f449" (UID: "70eced92-3511-4c69-9b83-d599f3e0f449"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:50:49.219976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.219940 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70eced92-3511-4c69-9b83-d599f3e0f449-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.219976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.219973 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.219976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.219982 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.220187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.219991 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkglr\" (UniqueName: \"kubernetes.io/projected/70eced92-3511-4c69-9b83-d599f3e0f449-kube-api-access-fkglr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.220187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.220001 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.220187 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.220009 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70eced92-3511-4c69-9b83-d599f3e0f449-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:50:49.516274 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.516236 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr"] Nov 03 00:50:49.519656 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.519624 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-xprnr"] Nov 03 00:50:49.997755 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:49.997726 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" path="/var/lib/kubelet/pods/70eced92-3511-4c69-9b83-d599f3e0f449/volumes" Nov 03 00:50:54.346000 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:54.345972 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:54.359049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:54.359021 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:50:55.158340 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:50:55.158301 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:51:01.966989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.966955 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j"] Nov 03 00:51:01.967975 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.967956 5144 cpu_manager.go:414] "containerMap: removing container" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" Nov 03 00:51:01.968061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.967978 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" Nov 03 00:51:01.968196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.968177 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="70eced92-3511-4c69-9b83-d599f3e0f449" containerName="istio-proxy" Nov 03 00:51:01.973273 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.973252 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:01.988070 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.988046 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:01.988238 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.988224 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:01.988314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.988263 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:01.988314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.988292 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:01.988436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.988339 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srrzr\" (UniqueName: \"kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:01.988436 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:01.988385 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.028445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.028418 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j"] Nov 03 00:51:02.073173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.073146 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c39710\"/\"explicit-model-c39710-kserve-self-signed-certs\"" Nov 03 00:51:02.084907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.084879 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-c39710\"/\"default-dockercfg-dbx2t\"" Nov 03 00:51:02.085054 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.084972 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c39710\"/\"kube-root-ca.crt\"" Nov 03 00:51:02.085115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.085072 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-c39710\"/\"openshift-service-ca.crt\"" Nov 03 00:51:02.089495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089473 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.089670 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089573 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.089756 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089729 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.089848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089830 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.089933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089867 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.089933 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089900 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srrzr\" (UniqueName: \"kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.090045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.089930 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.090133 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.090107 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.090454 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.090431 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.092551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.092531 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.092958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.092939 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.147026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.146994 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srrzr\" (UniqueName: \"kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr\") pod \"explicit-model-c39710-kserve-f4dd8785c-mts9j\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.285801 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.285772 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:02.665961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:02.665936 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j"] Nov 03 00:51:02.666988 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:51:02.666963 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod141c11c1_64db_47f4_9c44_58fd48f1b2e6.slice/crio-28c381dd09d75365a58e8df026602d064b6abd6354c7a49152da28e03392484d WatchSource:0}: Error finding container 28c381dd09d75365a58e8df026602d064b6abd6354c7a49152da28e03392484d: Status 404 returned error can't find the container with id 28c381dd09d75365a58e8df026602d064b6abd6354c7a49152da28e03392484d Nov 03 00:51:03.215225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:03.215184 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" event={"ID":"141c11c1-64db-47f4-9c44-58fd48f1b2e6","Type":"ContainerStarted","Data":"772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9"} Nov 03 00:51:03.215611 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:03.215235 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" event={"ID":"141c11c1-64db-47f4-9c44-58fd48f1b2e6","Type":"ContainerStarted","Data":"28c381dd09d75365a58e8df026602d064b6abd6354c7a49152da28e03392484d"} Nov 03 00:51:03.286455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:03.286426 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:03.329294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:03.329246 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" podStartSLOduration=2.329233668 podStartE2EDuration="2.329233668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:03.328301456 +0000 UTC m=+2753.473995928" watchObservedRunningTime="2026-11-03 00:51:03.329233668 +0000 UTC m=+2753.474928124" Nov 03 00:51:12.161345 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:51:12.161307 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:51:12.285983 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.285955 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:12.300512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.300489 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:12.313427 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.313404 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:51:12.508799 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.508773 5144 scope.go:122] "RemoveContainer" containerID="0b907d3dbe6a98c80c8b813840495cf5d1392cc436bd88d6b177760e15a38798" Nov 03 00:51:12.521147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.521119 5144 scope.go:122] "RemoveContainer" containerID="729778112f0e6b665602313223855030be1af0e53986285c3a5d923fb39b39d3" Nov 03 00:51:12.535767 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.535743 5144 scope.go:122] "RemoveContainer" containerID="f75b8bf9bd35169df6061ed2c56fa08a2def105003ab51d7391ca0deb3d621c2" Nov 03 00:51:12.551046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.551022 5144 scope.go:122] "RemoveContainer" containerID="5fb1fc87f8bbf715491c659ac52bc8a551119ff9d6499011705767a4f4a2df55" Nov 03 00:51:12.570388 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.570364 5144 scope.go:122] "RemoveContainer" containerID="450838d0a1b3fed8967bb4ec4b4dbd0ada622b414d2f988e436b886052dd450c" Nov 03 00:51:12.579921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.579901 5144 scope.go:122] "RemoveContainer" containerID="37f6e877eda9b637049524010527542426757dd9b1f8e98a4b8e0c49da085ba3" Nov 03 00:51:12.588901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:12.588884 5144 scope.go:122] "RemoveContainer" containerID="8d7a4220f4b4ae99e7e90ee38f7424f93534e8a7ed31d09da0d69a3c67bd3bc8" Nov 03 00:51:42.157585 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:51:42.157542 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:51:46.835895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:51:46.835862 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j"] Nov 03 00:52:01.858809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:01.858765 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" podUID="141c11c1-64db-47f4-9c44-58fd48f1b2e6" containerName="main" containerID="cri-o://772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9" gracePeriod=45 Nov 03 00:52:02.140015 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.139988 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:52:02.249284 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249250 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs\" (UniqueName: \"kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs\") pod \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " Nov 03 00:52:02.249474 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249453 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm\") pod \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " Nov 03 00:52:02.249531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249496 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home\") pod \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " Nov 03 00:52:02.249531 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249524 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache\") pod \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " Nov 03 00:52:02.249689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249557 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr\" (UniqueName: \"kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr\") pod \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " Nov 03 00:52:02.249689 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249607 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir\") pod \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\" (UID: \"141c11c1-64db-47f4-9c44-58fd48f1b2e6\") " Nov 03 00:52:02.249888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.249829 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache" pod "141c11c1-64db-47f4-9c44-58fd48f1b2e6" (UID: "141c11c1-64db-47f4-9c44-58fd48f1b2e6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:02.250087 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.250063 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir" pod "141c11c1-64db-47f4-9c44-58fd48f1b2e6" (UID: "141c11c1-64db-47f4-9c44-58fd48f1b2e6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:02.250372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.250352 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home" pod "141c11c1-64db-47f4-9c44-58fd48f1b2e6" (UID: "141c11c1-64db-47f4-9c44-58fd48f1b2e6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:02.250508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.250406 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:02.250623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.250517 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:02.252025 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.251998 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm" pod "141c11c1-64db-47f4-9c44-58fd48f1b2e6" (UID: "141c11c1-64db-47f4-9c44-58fd48f1b2e6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:02.252115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.252101 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs" pod "141c11c1-64db-47f4-9c44-58fd48f1b2e6" (UID: "141c11c1-64db-47f4-9c44-58fd48f1b2e6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:52:02.252218 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.252193 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr" pod "141c11c1-64db-47f4-9c44-58fd48f1b2e6" (UID: "141c11c1-64db-47f4-9c44-58fd48f1b2e6"). InnerVolumeSpecName "kube-api-access-srrzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:02.352314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.352278 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:02.352314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.352309 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141c11c1-64db-47f4-9c44-58fd48f1b2e6-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:02.352314 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.352321 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srrzr\" (UniqueName: \"kubernetes.io/projected/141c11c1-64db-47f4-9c44-58fd48f1b2e6-kube-api-access-srrzr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:02.352572 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.352331 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141c11c1-64db-47f4-9c44-58fd48f1b2e6-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:02.695962 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.695932 5144 generic.go:356] "Generic (PLEG): container finished" podID="141c11c1-64db-47f4-9c44-58fd48f1b2e6" containerID="772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9" exitCode=0 Nov 03 00:52:02.696153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.695974 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" Nov 03 00:52:02.696153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.695988 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" event={"ID":"141c11c1-64db-47f4-9c44-58fd48f1b2e6","Type":"ContainerDied","Data":"772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9"} Nov 03 00:52:02.696153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.696023 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j" event={"ID":"141c11c1-64db-47f4-9c44-58fd48f1b2e6","Type":"ContainerDied","Data":"28c381dd09d75365a58e8df026602d064b6abd6354c7a49152da28e03392484d"} Nov 03 00:52:02.696153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.696043 5144 scope.go:122] "RemoveContainer" containerID="772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9" Nov 03 00:52:02.707620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.707601 5144 scope.go:122] "RemoveContainer" containerID="772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9" Nov 03 00:52:02.707987 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:02.707968 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9\": container with ID starting with 772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9 not found: ID does not exist" containerID="772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9" Nov 03 00:52:02.708047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.707997 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9"} err="failed to get container status \"772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9\": rpc error: code = NotFound desc = could not find container \"772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9\": container with ID starting with 772eb1cec239b6971f3c4ce1109b2e63ec265633bdff6d33b6d43c85d75075f9 not found: ID does not exist" Nov 03 00:52:02.861749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.861711 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j"] Nov 03 00:52:02.870288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:02.870252 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-c39710/explicit-model-c39710-kserve-f4dd8785c-mts9j"] Nov 03 00:52:03.997803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:03.997772 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141c11c1-64db-47f4-9c44-58fd48f1b2e6" path="/var/lib/kubelet/pods/141c11c1-64db-47f4-9c44-58fd48f1b2e6/volumes" Nov 03 00:52:06.469859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:06.469822 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf"] Nov 03 00:52:06.470463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:06.470114 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" containerID="cri-o://35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02" gracePeriod=30 Nov 03 00:52:10.523843 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:10.523801 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 00:52:10.524263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:10.523891 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 00:52:11.755988 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.755961 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:52:11.771553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.771521 5144 generic.go:356] "Generic (PLEG): container finished" podID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerID="35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02" exitCode=0 Nov 03 00:52:11.771753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.771575 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" Nov 03 00:52:11.771753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.771603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" event={"ID":"bcb458a4-e6dd-41c8-84e9-da59100fcb49","Type":"ContainerDied","Data":"35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02"} Nov 03 00:52:11.771753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.771687 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf" event={"ID":"bcb458a4-e6dd-41c8-84e9-da59100fcb49","Type":"ContainerDied","Data":"a6ea1ec56b5592966c8e48c4fa5737231dd6b7fd655fa29d2b4cd9e7792ecac1"} Nov 03 00:52:11.771753 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.771717 5144 scope.go:122] "RemoveContainer" containerID="35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02" Nov 03 00:52:11.782746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.782726 5144 scope.go:122] "RemoveContainer" containerID="35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02" Nov 03 00:52:11.783014 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:11.782985 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02\": container with ID starting with 35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02 not found: ID does not exist" containerID="35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02" Nov 03 00:52:11.783094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.783018 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02"} err="failed to get container status \"35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02\": rpc error: code = NotFound desc = could not find container \"35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02\": container with ID starting with 35e1ed8262fdf2ce6ebfdc20586a6785ec084bdea2741fe6669dd14659bb1b02 not found: ID does not exist" Nov 03 00:52:11.880163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880134 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880219 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880261 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880286 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880311 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880360 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880349 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880397 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880380 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:11.880663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880546 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880606 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo\") pod \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\" (UID: \"bcb458a4-e6dd-41c8-84e9-da59100fcb49\") " Nov 03 00:52:11.880663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880547 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:11.880942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880686 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:11.880942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880810 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:11.881044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.880962 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:11.881487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.881462 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.881694 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.881669 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.881820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.881696 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.881820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.881709 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.881820 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.881722 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.882955 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.882931 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "kube-api-access-tdcmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:11.883116 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.883100 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:11.883412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.883389 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:52:11.883483 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.883405 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token" pod "bcb458a4-e6dd-41c8-84e9-da59100fcb49" (UID: "bcb458a4-e6dd-41c8-84e9-da59100fcb49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:11.982859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.982821 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdcmt\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-kube-api-access-tdcmt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.982859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.982852 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.982859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.982865 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:11.983103 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:11.982879 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcb458a4-e6dd-41c8-84e9-da59100fcb49-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:12.157064 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:12.156985 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:52:12.188545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:12.188514 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf"] Nov 03 00:52:12.194616 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:12.194588 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-vxsdf"] Nov 03 00:52:12.661941 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:12.661909 5144 scope.go:122] "RemoveContainer" containerID="5a9e7a7202965ee91513d6ef96e09940be808b8ee6599e840bbc5457fbf05fb3" Nov 03 00:52:13.997788 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:13.997754 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" path="/var/lib/kubelet/pods/bcb458a4-e6dd-41c8-84e9-da59100fcb49/volumes" Nov 03 00:52:14.905905 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.905872 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf"] Nov 03 00:52:14.906547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.906532 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" Nov 03 00:52:14.906595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.906549 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" Nov 03 00:52:14.906595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.906574 5144 cpu_manager.go:414] "containerMap: removing container" podUID="141c11c1-64db-47f4-9c44-58fd48f1b2e6" containerName="main" Nov 03 00:52:14.906595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.906579 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141c11c1-64db-47f4-9c44-58fd48f1b2e6" containerName="main" Nov 03 00:52:14.906710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.906685 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcb458a4-e6dd-41c8-84e9-da59100fcb49" containerName="istio-proxy" Nov 03 00:52:14.906710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.906699 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="141c11c1-64db-47f4-9c44-58fd48f1b2e6" containerName="main" Nov 03 00:52:14.911977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.911959 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:14.966881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:14.966850 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf"] Nov 03 00:52:15.011142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.011116 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c87e2f\"/\"auth-test-model-c87e2f-kserve-self-signed-certs\"" Nov 03 00:52:15.013189 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.013170 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c87e2f\"/\"kube-root-ca.crt\"" Nov 03 00:52:15.013445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.013427 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-c87e2f\"/\"openshift-service-ca.crt\"" Nov 03 00:52:15.013499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.013461 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-c87e2f\"/\"default-dockercfg-jktv9\"" Nov 03 00:52:15.018200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.018181 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.018305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.018205 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.018305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.018228 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.018305 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.018256 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.018450 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.018434 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8t76\" (UniqueName: \"kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.018502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.018490 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120071 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120040 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8t76\" (UniqueName: \"kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120134 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120239 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120254 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120286 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120319 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120717 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.120870 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.120822 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.121326 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.121302 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.122846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.122826 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.123026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.123009 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.177056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.176992 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8t76\" (UniqueName: \"kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76\") pod \"auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.225148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.225123 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:15.605004 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.604979 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf"] Nov 03 00:52:15.606022 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:52:15.606000 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod891cc89c_7f89_4b9f_bea2_bb515e50e67d.slice/crio-5b939a256df52a9714240799997cc2b16007bad62c397e9bd85163916f23de9c WatchSource:0}: Error finding container 5b939a256df52a9714240799997cc2b16007bad62c397e9bd85163916f23de9c: Status 404 returned error can't find the container with id 5b939a256df52a9714240799997cc2b16007bad62c397e9bd85163916f23de9c Nov 03 00:52:15.803404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.803372 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" event={"ID":"891cc89c-7f89-4b9f-bea2-bb515e50e67d","Type":"ContainerStarted","Data":"9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805"} Nov 03 00:52:15.803602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.803421 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" event={"ID":"891cc89c-7f89-4b9f-bea2-bb515e50e67d","Type":"ContainerStarted","Data":"5b939a256df52a9714240799997cc2b16007bad62c397e9bd85163916f23de9c"} Nov 03 00:52:15.926279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:15.926178 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" podStartSLOduration=1.926164448 podStartE2EDuration="1.926164448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:15.925296229 +0000 UTC m=+2826.070990682" watchObservedRunningTime="2026-11-03 00:52:15.926164448 +0000 UTC m=+2826.071858902" Nov 03 00:52:16.226222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:16.226152 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:24.515990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:24.515954 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs"] Nov 03 00:52:24.625743 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:24.625712 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5"] Nov 03 00:52:25.226093 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:25.226063 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:25.239742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:25.239721 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:25.900957 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:25.900926 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:52:29.516644 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.516573 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" podUID="41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" containerName="payload-processing" containerID="cri-o://899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c" gracePeriod=25 Nov 03 00:52:29.626579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.626530 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" podUID="dbb67e56-66d4-42aa-9bc2-d820c2a85e96" containerName="payload-pre-processing" containerID="cri-o://141a438131395705e2f1e6c617c61c2c6e8f13ff48633c74fdf34ca57cb63afb" gracePeriod=25 Nov 03 00:52:29.840209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.840180 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:52:29.904899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.904199 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume\") pod \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " Nov 03 00:52:29.904899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.904287 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42\" (UniqueName: \"kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42\") pod \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\" (UID: \"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5\") " Nov 03 00:52:29.905947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.905902 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume" pod "41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" (UID: "41df7b0e-61a2-4d06-8557-b75e9e7ac0a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:29.908380 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.908355 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42" pod "41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" (UID: "41df7b0e-61a2-4d06-8557-b75e9e7ac0a5"). InnerVolumeSpecName "kube-api-access-p8k42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:29.921806 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.921777 5144 generic.go:356] "Generic (PLEG): container finished" podID="dbb67e56-66d4-42aa-9bc2-d820c2a85e96" containerID="141a438131395705e2f1e6c617c61c2c6e8f13ff48633c74fdf34ca57cb63afb" exitCode=0 Nov 03 00:52:29.921938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.921853 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" event={"ID":"dbb67e56-66d4-42aa-9bc2-d820c2a85e96","Type":"ContainerDied","Data":"141a438131395705e2f1e6c617c61c2c6e8f13ff48633c74fdf34ca57cb63afb"} Nov 03 00:52:29.923619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.923599 5144 generic.go:356] "Generic (PLEG): container finished" podID="41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" containerID="899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c" exitCode=0 Nov 03 00:52:29.923766 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.923749 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" event={"ID":"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5","Type":"ContainerDied","Data":"899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c"} Nov 03 00:52:29.923816 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.923792 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" event={"ID":"41df7b0e-61a2-4d06-8557-b75e9e7ac0a5","Type":"ContainerDied","Data":"5af2b8e1c101ee5d7d2a4bdfe768ced6a11a7f2a33c36a0982a3027f303426d0"} Nov 03 00:52:29.923854 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.923814 5144 scope.go:122] "RemoveContainer" containerID="899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c" Nov 03 00:52:29.923950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.923941 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs" Nov 03 00:52:29.942295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.942272 5144 scope.go:122] "RemoveContainer" containerID="899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c" Nov 03 00:52:29.942599 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:29.942572 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c\": container with ID starting with 899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c not found: ID does not exist" containerID="899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c" Nov 03 00:52:29.942718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.942604 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c"} err="failed to get container status \"899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c\": rpc error: code = NotFound desc = could not find container \"899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c\": container with ID starting with 899d15f8191b9b7a5644450c3b533cc850ac093620b2254c33c7a5c4dc79c40c not found: ID does not exist" Nov 03 00:52:29.943311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:29.943298 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:52:30.009668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.006048 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume\") pod \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " Nov 03 00:52:30.009668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.006120 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46\" (UniqueName: \"kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46\") pod \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\" (UID: \"dbb67e56-66d4-42aa-9bc2-d820c2a85e96\") " Nov 03 00:52:30.009668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.006975 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:30.009668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.007002 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8k42\" (UniqueName: \"kubernetes.io/projected/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5-kube-api-access-p8k42\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:30.009668 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.007851 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume" pod "dbb67e56-66d4-42aa-9bc2-d820c2a85e96" (UID: "dbb67e56-66d4-42aa-9bc2-d820c2a85e96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:30.011595 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.011564 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46" pod "dbb67e56-66d4-42aa-9bc2-d820c2a85e96" (UID: "dbb67e56-66d4-42aa-9bc2-d820c2a85e96"). InnerVolumeSpecName "kube-api-access-txb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:30.108170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.108087 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:30.108170 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.108113 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txb46\" (UniqueName: \"kubernetes.io/projected/dbb67e56-66d4-42aa-9bc2-d820c2a85e96-kube-api-access-txb46\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:30.191438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.191406 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs"] Nov 03 00:52:30.196279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.196255 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-c39710-7494d5d59f-bvkfs"] Nov 03 00:52:30.933247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.933161 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" event={"ID":"dbb67e56-66d4-42aa-9bc2-d820c2a85e96","Type":"ContainerDied","Data":"7661c733b7b4888ba6ee65312a2b2ab79f9130d74873174e02b6342d9135b5e1"} Nov 03 00:52:30.933247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.933189 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5" Nov 03 00:52:30.933247 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:30.933214 5144 scope.go:122] "RemoveContainer" containerID="141a438131395705e2f1e6c617c61c2c6e8f13ff48633c74fdf34ca57cb63afb" Nov 03 00:52:31.097885 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.097852 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5"] Nov 03 00:52:31.107002 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.106975 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-c39710-7f744c595-mhkf5"] Nov 03 00:52:31.346275 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.346240 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft"] Nov 03 00:52:31.347705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.347677 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dbb67e56-66d4-42aa-9bc2-d820c2a85e96" containerName="payload-pre-processing" Nov 03 00:52:31.347705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.347704 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbb67e56-66d4-42aa-9bc2-d820c2a85e96" containerName="payload-pre-processing" Nov 03 00:52:31.347888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.347724 5144 cpu_manager.go:414] "containerMap: removing container" podUID="41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" containerName="payload-processing" Nov 03 00:52:31.347888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.347733 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" containerName="payload-processing" Nov 03 00:52:31.348084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.348067 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" containerName="payload-processing" Nov 03 00:52:31.348149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.348120 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbb67e56-66d4-42aa-9bc2-d820c2a85e96" containerName="payload-pre-processing" Nov 03 00:52:31.354132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.354112 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.401907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.401880 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft"] Nov 03 00:52:31.411202 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.411178 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"default-dockercfg-twj22\"" Nov 03 00:52:31.411369 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.411204 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"auth-test-model-20f0a3-kserve-self-signed-certs\"" Nov 03 00:52:31.411434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.411395 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"openshift-service-ca.crt\"" Nov 03 00:52:31.411538 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.411523 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20f0a3\"/\"kube-root-ca.crt\"" Nov 03 00:52:31.430350 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.430317 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.430456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.430369 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.430456 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.430438 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.430569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.430459 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.430692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.430672 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fpql\" (UniqueName: \"kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.430747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.430734 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532013 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.531973 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fpql\" (UniqueName: \"kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532026 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532124 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532165 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532215 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532238 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532497 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532520 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.532603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.532567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.534698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.534674 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.534853 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.534834 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.590793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.590768 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fpql\" (UniqueName: \"kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql\") pod \"auth-test-model-20f0a3-kserve-66d967cd5-7c5ft\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.667258 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.667196 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:31.996678 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.996594 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41df7b0e-61a2-4d06-8557-b75e9e7ac0a5" path="/var/lib/kubelet/pods/41df7b0e-61a2-4d06-8557-b75e9e7ac0a5/volumes" Nov 03 00:52:31.997051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:31.997034 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbb67e56-66d4-42aa-9bc2-d820c2a85e96" path="/var/lib/kubelet/pods/dbb67e56-66d4-42aa-9bc2-d820c2a85e96/volumes" Nov 03 00:52:32.045119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:32.045096 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft"] Nov 03 00:52:32.045404 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:52:32.045382 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod397f9a92_52ab_41b0_9e59_391189e7e04e.slice/crio-14034b5af441343073865a9a03d290e85df61a9feac555b83dfabb53bb509979 WatchSource:0}: Error finding container 14034b5af441343073865a9a03d290e85df61a9feac555b83dfabb53bb509979: Status 404 returned error can't find the container with id 14034b5af441343073865a9a03d290e85df61a9feac555b83dfabb53bb509979 Nov 03 00:52:32.953221 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:32.953185 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" event={"ID":"397f9a92-52ab-41b0-9e59-391189e7e04e","Type":"ContainerStarted","Data":"1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53"} Nov 03 00:52:32.953412 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:32.953234 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" event={"ID":"397f9a92-52ab-41b0-9e59-391189e7e04e","Type":"ContainerStarted","Data":"14034b5af441343073865a9a03d290e85df61a9feac555b83dfabb53bb509979"} Nov 03 00:52:33.067551 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:33.067502 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" podStartSLOduration=2.067488653 podStartE2EDuration="2.067488653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:33.067155332 +0000 UTC m=+2843.212849788" watchObservedRunningTime="2026-11-03 00:52:33.067488653 +0000 UTC m=+2843.213183108" Nov 03 00:52:33.667895 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:33.667865 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:35.048210 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:35.048174 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6"] Nov 03 00:52:35.066460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:35.066430 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a8ee86d-7zqb6"] Nov 03 00:52:35.999376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:35.999347 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ce3d85e-1853-4e62-938b-ccd54485dd78" path="/var/lib/kubelet/pods/8ce3d85e-1853-4e62-938b-ccd54485dd78/volumes" Nov 03 00:52:41.668358 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:41.668324 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:41.680588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:41.680562 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:42.061279 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:42.061245 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:52:42.157844 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:42.157806 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:52:46.046795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:46.046765 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd"] Nov 03 00:52:46.095718 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:46.095686 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d27b0c3e-qncpd"] Nov 03 00:52:46.664835 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:46.664802 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t"] Nov 03 00:52:46.665039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:46.665017 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" containerID="cri-o://2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9" gracePeriod=30 Nov 03 00:52:47.823349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.823316 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4"] Nov 03 00:52:47.827893 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.827871 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.937684 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqtn\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.937738 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.937792 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.937827 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.937915 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.937954 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.938002 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.938066 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.939621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.938160 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:47.940318 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.939876 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1f20dd-openshift-default-dockercfg-gczjk\"" Nov 03 00:52:47.972602 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.964382 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4"] Nov 03 00:52:47.999460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:47.999427 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee224846-7e83-4742-ac0d-d923e316391e" path="/var/lib/kubelet/pods/ee224846-7e83-4742-ac0d-d923e316391e/volumes" Nov 03 00:52:48.038951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.038922 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039032 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqtn\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039283 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039351 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039399 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039484 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039495 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039526 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039625 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039573 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039653 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039762 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039659 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.039977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.039952 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.040101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.040031 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.040383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.040343 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.042174 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.042154 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.042997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.042978 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.096115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.096051 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqtn\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.096444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.096426 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token\") pod \"e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.144286 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.144254 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:48.581734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.581710 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4"] Nov 03 00:52:48.582441 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:52:48.582415 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb56253a8_472f_44ec_a195_347e35a259d2.slice/crio-674f0854df0f0cb726336effb58360f97615abea3c06e97604b38502af0bcafa WatchSource:0}: Error finding container 674f0854df0f0cb726336effb58360f97615abea3c06e97604b38502af0bcafa: Status 404 returned error can't find the container with id 674f0854df0f0cb726336effb58360f97615abea3c06e97604b38502af0bcafa Nov 03 00:52:48.584873 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.584841 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:52:48.584963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.584901 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:52:48.584963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:48.584930 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:52:49.101464 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:49.101429 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" event={"ID":"b56253a8-472f-44ec-a195-347e35a259d2","Type":"ContainerStarted","Data":"5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2"} Nov 03 00:52:49.101867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:49.101490 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" event={"ID":"b56253a8-472f-44ec-a195-347e35a259d2","Type":"ContainerStarted","Data":"674f0854df0f0cb726336effb58360f97615abea3c06e97604b38502af0bcafa"} Nov 03 00:52:49.144936 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:49.144912 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:49.152584 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:49.152560 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:49.216091 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:49.216042 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" podStartSLOduration=2.216026303 podStartE2EDuration="2.216026303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:49.215464964 +0000 UTC m=+2859.361159421" watchObservedRunningTime="2026-11-03 00:52:49.216026303 +0000 UTC m=+2859.361720762" Nov 03 00:52:50.108487 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:50.108458 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:50.109485 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:50.109464 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:52:51.952607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:51.952537 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:52:52.089713 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.089685 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.089909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.089734 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.089909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.089775 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.089909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.089857 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.089909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.089901 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.090168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.089928 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.090168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090065 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:52.090168 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090106 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:52.090352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090172 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:52.090352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090206 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.090352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090256 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:52:52.090352 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090323 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.090749 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090389 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v\") pod \"35cfbf58-d149-459b-a6cc-ca200c99029f\" (UID: \"35cfbf58-d149-459b-a6cc-ca200c99029f\") " Nov 03 00:52:52.091148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.090937 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:52.091148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.091043 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.091148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.091064 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.091148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.091076 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35cfbf58-d149-459b-a6cc-ca200c99029f-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.091148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.091089 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.091148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.091102 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.092817 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.092791 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:52.093050 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.093022 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:52:52.093154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.093060 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:52:52.094090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.094063 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v" pod "35cfbf58-d149-459b-a6cc-ca200c99029f" (UID: "35cfbf58-d149-459b-a6cc-ca200c99029f"). InnerVolumeSpecName "kube-api-access-cbg7v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:52:52.126479 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.126453 5144 generic.go:356] "Generic (PLEG): container finished" podID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerID="2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9" exitCode=0 Nov 03 00:52:52.127265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.127224 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" event={"ID":"35cfbf58-d149-459b-a6cc-ca200c99029f","Type":"ContainerDied","Data":"2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9"} Nov 03 00:52:52.127367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.127262 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" Nov 03 00:52:52.127367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.127300 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t" event={"ID":"35cfbf58-d149-459b-a6cc-ca200c99029f","Type":"ContainerDied","Data":"fbd940cdfd1ab670733198eb8cadc9ef5d19b954397bed50e80f067846f044b8"} Nov 03 00:52:52.127367 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.127328 5144 scope.go:122] "RemoveContainer" containerID="2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9" Nov 03 00:52:52.144790 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.142828 5144 scope.go:122] "RemoveContainer" containerID="97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028" Nov 03 00:52:52.163786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.160374 5144 scope.go:122] "RemoveContainer" containerID="2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9" Nov 03 00:52:52.163786 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:52.160732 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9\": container with ID starting with 2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9 not found: ID does not exist" containerID="2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9" Nov 03 00:52:52.163786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.160765 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9"} err="failed to get container status \"2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9\": rpc error: code = NotFound desc = could not find container \"2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9\": container with ID starting with 2bff6d5cf679b49ee22ae2c21140726378a1185a839d11cfd02cfc675fd9fff9 not found: ID does not exist" Nov 03 00:52:52.163786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.160791 5144 scope.go:122] "RemoveContainer" containerID="97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028" Nov 03 00:52:52.163786 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:52:52.161072 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028\": container with ID starting with 97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028 not found: ID does not exist" containerID="97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028" Nov 03 00:52:52.163786 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.161102 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028"} err="failed to get container status \"97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028\": rpc error: code = NotFound desc = could not find container \"97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028\": container with ID starting with 97125bc3f71b07aea4900da9f3a52f73d73ea6f79e6ecb0fc25f0e4d53856028 not found: ID does not exist" Nov 03 00:52:52.192057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.192034 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.192182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.192061 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbg7v\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-kube-api-access-cbg7v\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.192182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.192075 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.192182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.192088 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35cfbf58-d149-459b-a6cc-ca200c99029f-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:52:52.291540 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.291513 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t"] Nov 03 00:52:52.297495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:52.297466 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-c39710-openshift-default-765c567d77-szb8t"] Nov 03 00:52:53.997362 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:52:53.997328 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" path="/var/lib/kubelet/pods/35cfbf58-d149-459b-a6cc-ca200c99029f/volumes" Nov 03 00:53:12.159449 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:53:12.159408 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:53:42.155706 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:53:42.155664 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:53:58.671217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:53:58.671185 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l"] Nov 03 00:53:58.671789 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:53:58.671378 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" containerID="cri-o://0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302" gracePeriod=30 Nov 03 00:54:03.951707 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:03.951641 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:54:04.033929 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.033905 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.033988 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034033 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034198 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034102 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034257 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034243 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034321 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034285 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034325 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034391 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034378 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034384 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:04.034550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034435 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo\") pod \"dfe14db8-a61e-4aa7-a314-c7e514099414\" (UID: \"dfe14db8-a61e-4aa7-a314-c7e514099414\") " Nov 03 00:54:04.034550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034435 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:04.034550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034439 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:04.034856 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.034606 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:04.035073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.035007 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.035073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.035031 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.035073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.035045 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.035073 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.035059 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfe14db8-a61e-4aa7-a314-c7e514099414-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.035705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.035679 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:04.036939 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.036902 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:04.037052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.036965 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:04.037121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.037098 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "kube-api-access-wfbmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:04.037235 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.037215 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo" pod "dfe14db8-a61e-4aa7-a314-c7e514099414" (UID: "dfe14db8-a61e-4aa7-a314-c7e514099414"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:54:04.136346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.136310 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfbmn\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-kube-api-access-wfbmn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.136346 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.136346 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.136543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.136360 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.136543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.136372 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.136543 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.136386 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfe14db8-a61e-4aa7-a314-c7e514099414-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:04.678679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.678621 5144 generic.go:356] "Generic (PLEG): container finished" podID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerID="0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302" exitCode=0 Nov 03 00:54:04.678995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.678675 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" event={"ID":"dfe14db8-a61e-4aa7-a314-c7e514099414","Type":"ContainerDied","Data":"0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302"} Nov 03 00:54:04.678995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.678695 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" Nov 03 00:54:04.678995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.678743 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l" event={"ID":"dfe14db8-a61e-4aa7-a314-c7e514099414","Type":"ContainerDied","Data":"98502ad68660b07667aeb42e6fae4b1f18a6ffcc7b98dbb1e709f264fa3320df"} Nov 03 00:54:04.678995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.678772 5144 scope.go:122] "RemoveContainer" containerID="0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302" Nov 03 00:54:04.692044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.691315 5144 scope.go:122] "RemoveContainer" containerID="0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302" Nov 03 00:54:04.692190 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:54:04.692164 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302\": container with ID starting with 0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302 not found: ID does not exist" containerID="0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302" Nov 03 00:54:04.692323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.692197 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302"} err="failed to get container status \"0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302\": rpc error: code = NotFound desc = could not find container \"0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302\": container with ID starting with 0c3bc720434faf7858182dd3794a35055f236c7b3a95c179a4fc615b175a6302 not found: ID does not exist" Nov 03 00:54:04.894822 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.894780 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l"] Nov 03 00:54:04.943131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:04.943063 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-pg29l"] Nov 03 00:54:05.996763 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:05.996729 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" path="/var/lib/kubelet/pods/dfe14db8-a61e-4aa7-a314-c7e514099414/volumes" Nov 03 00:54:12.156544 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:54:12.156503 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:54:12.729542 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:12.729512 5144 scope.go:122] "RemoveContainer" containerID="f131552bfd0b081db6cbd93736c2a3944f19da420132882fe7e1502be2f92161" Nov 03 00:54:12.741810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:12.741788 5144 scope.go:122] "RemoveContainer" containerID="b14b84c60438be0154a0380f19fa2c0cdab4dae9f4ad2b33c4c1b9cfefb6ea75" Nov 03 00:54:13.055738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:13.055710 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4"] Nov 03 00:54:13.055951 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:13.055923 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" podUID="b56253a8-472f-44ec-a195-347e35a259d2" containerName="istio-proxy" containerID="cri-o://5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2" gracePeriod=30 Nov 03 00:54:18.311887 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.311859 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:54:18.401842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.401775 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.401853 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.401901 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.401941 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.401963 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402008 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402033 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402094 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:18.402292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402105 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402246 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs\") pod \"b56253a8-472f-44ec-a195-347e35a259d2\" (UID: \"b56253a8-472f-44ec-a195-347e35a259d2\") " Nov 03 00:54:18.402292 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402252 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:18.402594 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402561 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:18.402683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402645 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:54:18.402750 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402694 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:18.402906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402885 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.402906 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402908 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.403081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402921 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b56253a8-472f-44ec-a195-347e35a259d2-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.403081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402935 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.403081 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.402949 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.404582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.404557 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 00:54:18.404795 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.404774 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:54:18.404878 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.404838 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:18.404938 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.404911 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn" pod "b56253a8-472f-44ec-a195-347e35a259d2" (UID: "b56253a8-472f-44ec-a195-347e35a259d2"). InnerVolumeSpecName "kube-api-access-mcqtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:54:18.504245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.504214 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.504245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.504243 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b56253a8-472f-44ec-a195-347e35a259d2-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.504245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.504252 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b56253a8-472f-44ec-a195-347e35a259d2-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.504462 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.504261 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqtn\" (UniqueName: \"kubernetes.io/projected/b56253a8-472f-44ec-a195-347e35a259d2-kube-api-access-mcqtn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:54:18.790079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.790044 5144 generic.go:356] "Generic (PLEG): container finished" podID="b56253a8-472f-44ec-a195-347e35a259d2" containerID="5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2" exitCode=0 Nov 03 00:54:18.790252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.790094 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" Nov 03 00:54:18.790252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.790134 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" event={"ID":"b56253a8-472f-44ec-a195-347e35a259d2","Type":"ContainerDied","Data":"5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2"} Nov 03 00:54:18.790252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.790197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4" event={"ID":"b56253a8-472f-44ec-a195-347e35a259d2","Type":"ContainerDied","Data":"674f0854df0f0cb726336effb58360f97615abea3c06e97604b38502af0bcafa"} Nov 03 00:54:18.790252 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.790221 5144 scope.go:122] "RemoveContainer" containerID="5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2" Nov 03 00:54:18.806683 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.806656 5144 scope.go:122] "RemoveContainer" containerID="5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2" Nov 03 00:54:18.807035 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:54:18.807015 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2\": container with ID starting with 5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2 not found: ID does not exist" containerID="5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2" Nov 03 00:54:18.807110 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.807042 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2"} err="failed to get container status \"5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2\": rpc error: code = NotFound desc = could not find container \"5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2\": container with ID starting with 5ec1c3d6ef3a452fe08c7b43a48d23a222cddce4fe363268e59bf93fde5a91e2 not found: ID does not exist" Nov 03 00:54:18.954516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:18.954487 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4"] Nov 03 00:54:19.004129 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:19.004099 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1f20dd-openshift-default-7f4bd4c45f-fjgq4"] Nov 03 00:54:19.996525 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:54:19.996481 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b56253a8-472f-44ec-a195-347e35a259d2" path="/var/lib/kubelet/pods/b56253a8-472f-44ec-a195-347e35a259d2/volumes" Nov 03 00:54:42.157272 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:54:42.157227 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:55:07.181619 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.181585 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h"] Nov 03 00:55:07.182242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182227 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b56253a8-472f-44ec-a195-347e35a259d2" containerName="istio-proxy" Nov 03 00:55:07.182299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182243 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b56253a8-472f-44ec-a195-347e35a259d2" containerName="istio-proxy" Nov 03 00:55:07.182299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182264 5144 cpu_manager.go:414] "containerMap: removing container" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" Nov 03 00:55:07.182299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182270 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" Nov 03 00:55:07.182299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182290 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" Nov 03 00:55:07.182299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182295 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" Nov 03 00:55:07.182469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182412 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b56253a8-472f-44ec-a195-347e35a259d2" containerName="istio-proxy" Nov 03 00:55:07.182469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182420 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" Nov 03 00:55:07.182469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182430 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" Nov 03 00:55:07.182469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.182437 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfe14db8-a61e-4aa7-a314-c7e514099414" containerName="istio-proxy" Nov 03 00:55:07.186173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.186156 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.241262 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.241232 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h"] Nov 03 00:55:07.286194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.286171 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:55:07.286325 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.286216 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:55:07.299295 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.299272 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.299398 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.299336 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.299463 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.299448 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2tff\" (UniqueName: \"kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.400883 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.400860 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2tff\" (UniqueName: \"kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.401003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.400936 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.401046 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.401010 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.401596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.401572 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.403311 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.403285 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.457778 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.457727 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2tff\" (UniqueName: \"kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff\") pod \"maas-api-revoke-sub-a6886ed5-dsb4h\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.500529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.500506 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:07.875160 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:07.875131 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h"] Nov 03 00:55:07.877003 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:55:07.876972 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cac8548_13a9_43ce_ba21_2c1676a5c76d.slice/crio-8af10c94840c31a5f6fa09ecadfe963954671360fe8e1c4cea0e70377c605202 WatchSource:0}: Error finding container 8af10c94840c31a5f6fa09ecadfe963954671360fe8e1c4cea0e70377c605202: Status 404 returned error can't find the container with id 8af10c94840c31a5f6fa09ecadfe963954671360fe8e1c4cea0e70377c605202 Nov 03 00:55:08.165780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:08.165745 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" event={"ID":"8cac8548-13a9-43ce-ba21-2c1676a5c76d","Type":"ContainerStarted","Data":"8af10c94840c31a5f6fa09ecadfe963954671360fe8e1c4cea0e70377c605202"} Nov 03 00:55:09.176365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:09.176279 5144 generic.go:356] "Generic (PLEG): container finished" podID="8cac8548-13a9-43ce-ba21-2c1676a5c76d" containerID="564f6860f93c9baa4c1791f83dea200a576fe325b33f53d6126af0e63c4091c1" exitCode=0 Nov 03 00:55:09.176365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:09.176324 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" event={"ID":"8cac8548-13a9-43ce-ba21-2c1676a5c76d","Type":"ContainerDied","Data":"564f6860f93c9baa4c1791f83dea200a576fe325b33f53d6126af0e63c4091c1"} Nov 03 00:55:10.347261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.347235 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:10.433118 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.433092 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca\") pod \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " Nov 03 00:55:10.433246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.433185 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff\" (UniqueName: \"kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff\") pod \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " Nov 03 00:55:10.433246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.433214 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca\") pod \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\" (UID: \"8cac8548-13a9-43ce-ba21-2c1676a5c76d\") " Nov 03 00:55:10.433663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.433578 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca" pod "8cac8548-13a9-43ce-ba21-2c1676a5c76d" (UID: "8cac8548-13a9-43ce-ba21-2c1676a5c76d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:10.435312 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.435285 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca" pod "8cac8548-13a9-43ce-ba21-2c1676a5c76d" (UID: "8cac8548-13a9-43ce-ba21-2c1676a5c76d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:10.435497 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.435481 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff" pod "8cac8548-13a9-43ce-ba21-2c1676a5c76d" (UID: "8cac8548-13a9-43ce-ba21-2c1676a5c76d"). InnerVolumeSpecName "kube-api-access-h2tff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:10.534590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.534555 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:10.534590 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.534589 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2tff\" (UniqueName: \"kubernetes.io/projected/8cac8548-13a9-43ce-ba21-2c1676a5c76d-kube-api-access-h2tff\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:10.534821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.534604 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cac8548-13a9-43ce-ba21-2c1676a5c76d-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:10.677233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.677207 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:55:10.688508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:10.688444 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 00:55:11.195242 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.195208 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" event={"ID":"8cac8548-13a9-43ce-ba21-2c1676a5c76d","Type":"ContainerDied","Data":"8af10c94840c31a5f6fa09ecadfe963954671360fe8e1c4cea0e70377c605202"} Nov 03 00:55:11.195421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.195248 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h" Nov 03 00:55:11.195421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.195256 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8af10c94840c31a5f6fa09ecadfe963954671360fe8e1c4cea0e70377c605202" Nov 03 00:55:11.348593 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.348560 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz"] Nov 03 00:55:11.366654 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.350065 5144 cpu_manager.go:414] "containerMap: removing container" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" Nov 03 00:55:11.366921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.366883 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35cfbf58-d149-459b-a6cc-ca200c99029f" containerName="istio-proxy" Nov 03 00:55:11.367516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.366942 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8cac8548-13a9-43ce-ba21-2c1676a5c76d" containerName="revoke-keys" Nov 03 00:55:11.367516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.366952 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cac8548-13a9-43ce-ba21-2c1676a5c76d" containerName="revoke-keys" Nov 03 00:55:11.367516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.367233 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cac8548-13a9-43ce-ba21-2c1676a5c76d" containerName="revoke-keys" Nov 03 00:55:11.382468 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.381914 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.394240 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.394218 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz"] Nov 03 00:55:11.434269 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.434242 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-f40c6c-dockercfg-b9vqx\"" Nov 03 00:55:11.434569 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.434552 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-f40c6c\"" Nov 03 00:55:11.448111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.447775 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njgwz\" (UniqueName: \"kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz\") pod \"payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.448111 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.447958 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.507859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.507831 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42"] Nov 03 00:55:11.512072 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.512050 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.549333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.549305 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njgwz\" (UniqueName: \"kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz\") pod \"payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.549500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.549422 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.550164 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.550143 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.568132 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.568109 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42"] Nov 03 00:55:11.606928 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.606901 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njgwz\" (UniqueName: \"kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz\") pod \"payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.650553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.650515 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.650736 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.650699 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x29nm\" (UniqueName: \"kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm\") pod \"payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.702868 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.702785 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:11.753329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.752406 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.753329 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.753307 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x29nm\" (UniqueName: \"kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm\") pod \"payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.753570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.753467 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.816426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.816357 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x29nm\" (UniqueName: \"kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm\") pod \"payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:11.826342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:11.826130 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:12.096968 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:55:12.096937 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2102ad_09d9_4e6d_8948_f9b328b85b68.slice/crio-a83fdccccf66c91bb8337a38b52cc831f8bd4468da34fb9674b9129551ae283c WatchSource:0}: Error finding container a83fdccccf66c91bb8337a38b52cc831f8bd4468da34fb9674b9129551ae283c: Status 404 returned error can't find the container with id a83fdccccf66c91bb8337a38b52cc831f8bd4468da34fb9674b9129551ae283c Nov 03 00:55:12.098481 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:12.098454 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz"] Nov 03 00:55:12.178853 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:55:12.178809 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:55:12.208335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:12.208300 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" event={"ID":"2e2102ad-09d9-4e6d-8948-f9b328b85b68","Type":"ContainerStarted","Data":"bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3"} Nov 03 00:55:12.208455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:12.208378 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" event={"ID":"2e2102ad-09d9-4e6d-8948-f9b328b85b68","Type":"ContainerStarted","Data":"a83fdccccf66c91bb8337a38b52cc831f8bd4468da34fb9674b9129551ae283c"} Nov 03 00:55:12.208455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:12.208404 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:12.250069 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:12.250040 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42"] Nov 03 00:55:12.250480 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:55:12.250453 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb5eb9c_80db_4590_b366_343da3832c2a.slice/crio-5d2b9eb77ecb6f36cd0818bb3bd9c8cff755d7639489518cfc172689e457803f WatchSource:0}: Error finding container 5d2b9eb77ecb6f36cd0818bb3bd9c8cff755d7639489518cfc172689e457803f: Status 404 returned error can't find the container with id 5d2b9eb77ecb6f36cd0818bb3bd9c8cff755d7639489518cfc172689e457803f Nov 03 00:55:12.325056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:12.325005 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" podStartSLOduration=1.324990018 podStartE2EDuration="1.324990018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:12.319660664 +0000 UTC m=+3002.465355121" watchObservedRunningTime="2026-11-03 00:55:12.324990018 +0000 UTC m=+3002.470684492" Nov 03 00:55:13.219280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:13.219179 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" event={"ID":"dcb5eb9c-80db-4590-b366-343da3832c2a","Type":"ContainerStarted","Data":"f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4"} Nov 03 00:55:13.219280 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:13.219237 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" event={"ID":"dcb5eb9c-80db-4590-b366-343da3832c2a","Type":"ContainerStarted","Data":"5d2b9eb77ecb6f36cd0818bb3bd9c8cff755d7639489518cfc172689e457803f"} Nov 03 00:55:13.219942 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:13.219488 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:13.333797 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:13.333747 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" podStartSLOduration=2.3337342420000002 podStartE2EDuration="2.333734242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:13.330313432 +0000 UTC m=+3003.476007890" watchObservedRunningTime="2026-11-03 00:55:13.333734242 +0000 UTC m=+3003.479428696" Nov 03 00:55:17.920357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:17.920321 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx"] Nov 03 00:55:17.926153 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:17.926133 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:17.977846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:17.977818 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:55:17.977989 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:17.977909 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:55:17.980740 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:17.980712 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx"] Nov 03 00:55:18.030808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.030778 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6dxx\" (UniqueName: \"kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.030978 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.030939 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.031044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.030988 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.131663 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.131616 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6dxx\" (UniqueName: \"kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.131836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.131823 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.131907 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.131888 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.132785 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.132757 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.134601 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.134577 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.190007 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.189944 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6dxx\" (UniqueName: \"kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx\") pod \"maas-api-revoke-sub-6eb9923f-nwnjx\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.241216 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.241194 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:18.617952 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:18.617919 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx"] Nov 03 00:55:18.619944 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:55:18.619909 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb218b76_2d5e_4471_a2ca_d8888cb73b7c.slice/crio-8bb916d9f4872a495fbe59abfbc9187f2f73190d9b45b433410d4dda289170e4 WatchSource:0}: Error finding container 8bb916d9f4872a495fbe59abfbc9187f2f73190d9b45b433410d4dda289170e4: Status 404 returned error can't find the container with id 8bb916d9f4872a495fbe59abfbc9187f2f73190d9b45b433410d4dda289170e4 Nov 03 00:55:19.269384 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:19.269348 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" event={"ID":"eb218b76-2d5e-4471-a2ca-d8888cb73b7c","Type":"ContainerStarted","Data":"865ec5d2f572fad4e719e25f75be431a0c7a460cd672e74a893b6c78a3bae1cc"} Nov 03 00:55:19.269807 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:19.269412 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" event={"ID":"eb218b76-2d5e-4471-a2ca-d8888cb73b7c","Type":"ContainerStarted","Data":"8bb916d9f4872a495fbe59abfbc9187f2f73190d9b45b433410d4dda289170e4"} Nov 03 00:55:19.381991 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:19.381940 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" podStartSLOduration=2.381926426 podStartE2EDuration="2.381926426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:19.381015048 +0000 UTC m=+3009.526709522" watchObservedRunningTime="2026-11-03 00:55:19.381926426 +0000 UTC m=+3009.527620920" Nov 03 00:55:20.280489 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:20.280445 5144 generic.go:356] "Generic (PLEG): container finished" podID="eb218b76-2d5e-4471-a2ca-d8888cb73b7c" containerID="865ec5d2f572fad4e719e25f75be431a0c7a460cd672e74a893b6c78a3bae1cc" exitCode=0 Nov 03 00:55:20.280958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:20.280535 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" event={"ID":"eb218b76-2d5e-4471-a2ca-d8888cb73b7c","Type":"ContainerDied","Data":"865ec5d2f572fad4e719e25f75be431a0c7a460cd672e74a893b6c78a3bae1cc"} Nov 03 00:55:21.504408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.504384 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:21.577018 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.576930 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx\" (UniqueName: \"kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx\") pod \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " Nov 03 00:55:21.577200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.577028 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca\") pod \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " Nov 03 00:55:21.577200 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.577088 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca\") pod \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\" (UID: \"eb218b76-2d5e-4471-a2ca-d8888cb73b7c\") " Nov 03 00:55:21.577453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.577431 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca" pod "eb218b76-2d5e-4471-a2ca-d8888cb73b7c" (UID: "eb218b76-2d5e-4471-a2ca-d8888cb73b7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:55:21.579263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.579242 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca" pod "eb218b76-2d5e-4471-a2ca-d8888cb73b7c" (UID: "eb218b76-2d5e-4471-a2ca-d8888cb73b7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:55:21.579359 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.579289 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx" pod "eb218b76-2d5e-4471-a2ca-d8888cb73b7c" (UID: "eb218b76-2d5e-4471-a2ca-d8888cb73b7c"). InnerVolumeSpecName "kube-api-access-v6dxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:55:21.678446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.678407 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6dxx\" (UniqueName: \"kubernetes.io/projected/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-kube-api-access-v6dxx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:21.678446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.678441 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:21.678446 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:21.678453 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb218b76-2d5e-4471-a2ca-d8888cb73b7c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:55:22.297953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:22.297869 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" Nov 03 00:55:22.297953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:22.297897 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx" event={"ID":"eb218b76-2d5e-4471-a2ca-d8888cb73b7c","Type":"ContainerDied","Data":"8bb916d9f4872a495fbe59abfbc9187f2f73190d9b45b433410d4dda289170e4"} Nov 03 00:55:22.297953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:22.297942 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bb916d9f4872a495fbe59abfbc9187f2f73190d9b45b433410d4dda289170e4" Nov 03 00:55:23.222888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:23.222862 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 00:55:24.231308 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:24.231278 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 00:55:42.156849 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:55:42.156798 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:55:47.635051 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.635017 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p"] Nov 03 00:55:47.635944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.635922 5144 cpu_manager.go:414] "containerMap: removing container" podUID="eb218b76-2d5e-4471-a2ca-d8888cb73b7c" containerName="revoke-keys" Nov 03 00:55:47.636084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.635946 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb218b76-2d5e-4471-a2ca-d8888cb73b7c" containerName="revoke-keys" Nov 03 00:55:47.636244 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.636174 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb218b76-2d5e-4471-a2ca-d8888cb73b7c" containerName="revoke-keys" Nov 03 00:55:47.640976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.640954 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.700864 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.700828 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4tlr\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.700911 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.700949 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.700994 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.701078 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.701122 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701211 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.701186 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.701267 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.701371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.701320 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.741614 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.741579 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p"] Nov 03 00:55:47.754958 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.754926 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-115ee5-openshift-default-dockercfg-khvkf\"" Nov 03 00:55:47.811228 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.811192 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.812524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.812498 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.812722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.812706 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.812855 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.812843 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.812993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.812981 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.814191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.814173 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.814665 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.811890 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.814793 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.813791 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.814877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.813282 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.814953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.814518 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.815057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.815042 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4tlr\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.815457 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.815437 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.816241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.816223 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.816757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.816165 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.842413 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.838297 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.843609 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.843580 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.871148 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.871119 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.873126 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.873073 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4tlr\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr\") pod \"e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:47.957876 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:47.957785 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:48.395183 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:55:48.395152 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b09322_4337_4ff7_b0e4_bb3cc82436d5.slice/crio-810a9df64b1e9fff226d64fe007104f6fccf947a66cf1513318049a3af030089 WatchSource:0}: Error finding container 810a9df64b1e9fff226d64fe007104f6fccf947a66cf1513318049a3af030089: Status 404 returned error can't find the container with id 810a9df64b1e9fff226d64fe007104f6fccf947a66cf1513318049a3af030089 Nov 03 00:55:48.395375 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.395355 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p"] Nov 03 00:55:48.397586 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.397550 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:55:48.397757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.397667 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:55:48.397757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.397715 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:55:48.505123 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.505087 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" event={"ID":"51b09322-4337-4ff7-b0e4-bb3cc82436d5","Type":"ContainerStarted","Data":"8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b"} Nov 03 00:55:48.505324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.505155 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" event={"ID":"51b09322-4337-4ff7-b0e4-bb3cc82436d5","Type":"ContainerStarted","Data":"810a9df64b1e9fff226d64fe007104f6fccf947a66cf1513318049a3af030089"} Nov 03 00:55:48.626932 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.626872 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podStartSLOduration=1.626854132 podStartE2EDuration="1.626854132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:48.620418453 +0000 UTC m=+3038.766112918" watchObservedRunningTime="2026-11-03 00:55:48.626854132 +0000 UTC m=+3038.772548590" Nov 03 00:55:48.958692 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:48.958649 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:49.964488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:49.964460 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:50.520655 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:50.520606 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:50.521826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:50.521805 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:55:59.345576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.345541 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp"] Nov 03 00:55:59.350012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.349995 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.401812 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.401772 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:55:59.451521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.451494 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:55:59.456067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.456041 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp"] Nov 03 00:55:59.473757 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.473732 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.473901 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.473779 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnz2j\" (UniqueName: \"kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.473949 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.473926 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.575052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.575026 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.575254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.575075 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnz2j\" (UniqueName: \"kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.575254 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.575125 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.575877 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.575855 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.577622 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.577588 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.631576 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.631511 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnz2j\" (UniqueName: \"kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j\") pod \"maas-api-revoke-sub-f735f37a-rqjcp\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:55:59.665494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:55:59.665470 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:56:00.048313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:00.048285 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp"] Nov 03 00:56:00.049375 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:00.049342 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf2c13a_5ca2_4e72_a229_bb4747fef695.slice/crio-60fe98297c8018b83dcbd40e907a74b73660b6e5cdad9634148de13cc55ecc8e WatchSource:0}: Error finding container 60fe98297c8018b83dcbd40e907a74b73660b6e5cdad9634148de13cc55ecc8e: Status 404 returned error can't find the container with id 60fe98297c8018b83dcbd40e907a74b73660b6e5cdad9634148de13cc55ecc8e Nov 03 00:56:00.607915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:00.607826 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" event={"ID":"caf2c13a-5ca2-4e72-a229-bb4747fef695","Type":"ContainerStarted","Data":"9ef698e64a19b3b9ebfb2eff0be150c71cb15741f189818f92b1b9c7fe008a8d"} Nov 03 00:56:00.607915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:00.607912 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" event={"ID":"caf2c13a-5ca2-4e72-a229-bb4747fef695","Type":"ContainerStarted","Data":"60fe98297c8018b83dcbd40e907a74b73660b6e5cdad9634148de13cc55ecc8e"} Nov 03 00:56:00.720837 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:00.720790 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" podStartSLOduration=1.720777947 podStartE2EDuration="1.720777947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:00.720463742 +0000 UTC m=+3050.866158219" watchObservedRunningTime="2026-11-03 00:56:00.720777947 +0000 UTC m=+3050.866472403" Nov 03 00:56:01.616548 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:01.616513 5144 generic.go:356] "Generic (PLEG): container finished" podID="caf2c13a-5ca2-4e72-a229-bb4747fef695" containerID="9ef698e64a19b3b9ebfb2eff0be150c71cb15741f189818f92b1b9c7fe008a8d" exitCode=0 Nov 03 00:56:01.617008 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:01.616589 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" event={"ID":"caf2c13a-5ca2-4e72-a229-bb4747fef695","Type":"ContainerDied","Data":"9ef698e64a19b3b9ebfb2eff0be150c71cb15741f189818f92b1b9c7fe008a8d"} Nov 03 00:56:02.802960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.802936 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:56:02.914747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.914656 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j\" (UniqueName: \"kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j\") pod \"caf2c13a-5ca2-4e72-a229-bb4747fef695\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " Nov 03 00:56:02.914747 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.914727 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca\") pod \"caf2c13a-5ca2-4e72-a229-bb4747fef695\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " Nov 03 00:56:02.914980 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.914795 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca\") pod \"caf2c13a-5ca2-4e72-a229-bb4747fef695\" (UID: \"caf2c13a-5ca2-4e72-a229-bb4747fef695\") " Nov 03 00:56:02.915139 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.915110 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca" pod "caf2c13a-5ca2-4e72-a229-bb4747fef695" (UID: "caf2c13a-5ca2-4e72-a229-bb4747fef695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:02.915335 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.915311 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:02.917294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.917271 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j" pod "caf2c13a-5ca2-4e72-a229-bb4747fef695" (UID: "caf2c13a-5ca2-4e72-a229-bb4747fef695"). InnerVolumeSpecName "kube-api-access-tnz2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:02.917447 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:02.917424 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca" pod "caf2c13a-5ca2-4e72-a229-bb4747fef695" (UID: "caf2c13a-5ca2-4e72-a229-bb4747fef695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:03.016703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:03.016667 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caf2c13a-5ca2-4e72-a229-bb4747fef695-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:03.016703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:03.016700 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnz2j\" (UniqueName: \"kubernetes.io/projected/caf2c13a-5ca2-4e72-a229-bb4747fef695-kube-api-access-tnz2j\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:03.632323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:03.632288 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" Nov 03 00:56:03.632323 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:03.632305 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp" event={"ID":"caf2c13a-5ca2-4e72-a229-bb4747fef695","Type":"ContainerDied","Data":"60fe98297c8018b83dcbd40e907a74b73660b6e5cdad9634148de13cc55ecc8e"} Nov 03 00:56:03.632554 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:03.632348 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fe98297c8018b83dcbd40e907a74b73660b6e5cdad9634148de13cc55ecc8e" Nov 03 00:56:10.095903 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.095870 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq"] Nov 03 00:56:10.096842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.096821 5144 cpu_manager.go:414] "containerMap: removing container" podUID="caf2c13a-5ca2-4e72-a229-bb4747fef695" containerName="revoke-keys" Nov 03 00:56:10.096842 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.096841 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf2c13a-5ca2-4e72-a229-bb4747fef695" containerName="revoke-keys" Nov 03 00:56:10.097044 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.097032 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf2c13a-5ca2-4e72-a229-bb4747fef695" containerName="revoke-keys" Nov 03 00:56:10.101167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.101148 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.154424 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.154394 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq"] Nov 03 00:56:10.198844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.198819 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:56:10.199041 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.199023 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:56:10.207378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.207353 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.207478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.207437 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gks\" (UniqueName: \"kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.207478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.207472 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.308420 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.308389 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.308620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.308443 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gks\" (UniqueName: \"kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.308620 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.308487 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.309089 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.309064 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.310977 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.310951 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.364972 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.364906 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gks\" (UniqueName: \"kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks\") pod \"maas-api-revoke-sub-e3bb471c-q82fq\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.415610 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.415579 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:10.797141 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:10.797108 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq"] Nov 03 00:56:10.797371 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:10.797345 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc6852e6_ff27_4e7c_ac40_0399e7455cde.slice/crio-9582722ac15db61f43fd75cc9cd82630c647fad0a15997b405872c237c0958e2 WatchSource:0}: Error finding container 9582722ac15db61f43fd75cc9cd82630c647fad0a15997b405872c237c0958e2: Status 404 returned error can't find the container with id 9582722ac15db61f43fd75cc9cd82630c647fad0a15997b405872c237c0958e2 Nov 03 00:56:11.696648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:11.696605 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" event={"ID":"bc6852e6-ff27-4e7c-ac40-0399e7455cde","Type":"ContainerStarted","Data":"f61ee819ecde046c6ff06565655357570708209daa1b35e8c721691685ad1adb"} Nov 03 00:56:11.696982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:11.696682 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" event={"ID":"bc6852e6-ff27-4e7c-ac40-0399e7455cde","Type":"ContainerStarted","Data":"9582722ac15db61f43fd75cc9cd82630c647fad0a15997b405872c237c0958e2"} Nov 03 00:56:11.809604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:11.809544 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" podStartSLOduration=1.8095306230000001 podStartE2EDuration="1.809530623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:11.808897956 +0000 UTC m=+3061.954592424" watchObservedRunningTime="2026-11-03 00:56:11.809530623 +0000 UTC m=+3061.955225079" Nov 03 00:56:12.159832 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:56:12.159794 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:56:12.704652 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:12.704598 5144 generic.go:356] "Generic (PLEG): container finished" podID="bc6852e6-ff27-4e7c-ac40-0399e7455cde" containerID="f61ee819ecde046c6ff06565655357570708209daa1b35e8c721691685ad1adb" exitCode=0 Nov 03 00:56:12.705052 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:12.704663 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" event={"ID":"bc6852e6-ff27-4e7c-ac40-0399e7455cde","Type":"ContainerDied","Data":"f61ee819ecde046c6ff06565655357570708209daa1b35e8c721691685ad1adb"} Nov 03 00:56:13.909494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.909465 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:13.955947 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.955915 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca\") pod \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " Nov 03 00:56:13.956131 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.956101 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks\" (UniqueName: \"kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks\") pod \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " Nov 03 00:56:13.956179 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.956131 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca\") pod \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\" (UID: \"bc6852e6-ff27-4e7c-ac40-0399e7455cde\") " Nov 03 00:56:13.956270 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.956247 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca" pod "bc6852e6-ff27-4e7c-ac40-0399e7455cde" (UID: "bc6852e6-ff27-4e7c-ac40-0399e7455cde"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:13.956714 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.956689 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:13.958411 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.958378 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca" pod "bc6852e6-ff27-4e7c-ac40-0399e7455cde" (UID: "bc6852e6-ff27-4e7c-ac40-0399e7455cde"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:13.958521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:13.958447 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks" pod "bc6852e6-ff27-4e7c-ac40-0399e7455cde" (UID: "bc6852e6-ff27-4e7c-ac40-0399e7455cde"). InnerVolumeSpecName "kube-api-access-c4gks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:14.058035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:14.058004 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gks\" (UniqueName: \"kubernetes.io/projected/bc6852e6-ff27-4e7c-ac40-0399e7455cde-kube-api-access-c4gks\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:14.058035 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:14.058036 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc6852e6-ff27-4e7c-ac40-0399e7455cde-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:14.735579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:14.735547 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" event={"ID":"bc6852e6-ff27-4e7c-ac40-0399e7455cde","Type":"ContainerDied","Data":"9582722ac15db61f43fd75cc9cd82630c647fad0a15997b405872c237c0958e2"} Nov 03 00:56:14.735754 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:14.735582 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq" Nov 03 00:56:14.735754 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:14.735594 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9582722ac15db61f43fd75cc9cd82630c647fad0a15997b405872c237c0958e2" Nov 03 00:56:42.157145 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:56:42.157105 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:56:43.958149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:43.958113 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46"] Nov 03 00:56:43.958848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:43.958832 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bc6852e6-ff27-4e7c-ac40-0399e7455cde" containerName="revoke-keys" Nov 03 00:56:43.958899 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:43.958849 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc6852e6-ff27-4e7c-ac40-0399e7455cde" containerName="revoke-keys" Nov 03 00:56:43.958964 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:43.958955 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc6852e6-ff27-4e7c-ac40-0399e7455cde" containerName="revoke-keys" Nov 03 00:56:43.962592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:43.962571 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.028147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.028099 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc7jh\" (UniqueName: \"kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.028803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.028156 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.028803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.028507 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.064117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.064079 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:56:44.064324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.064167 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:56:44.070522 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.070490 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46"] Nov 03 00:56:44.129589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.129553 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.129815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.129686 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc7jh\" (UniqueName: \"kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.129815 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.129721 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.130402 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.130372 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.132337 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.132309 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.186193 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.186165 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc7jh\" (UniqueName: \"kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh\") pod \"maas-api-revoke-sub-039f22b6-nhj46\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.277948 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.277923 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:44.664238 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:44.664206 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5bdc82_0cc5_4278_b4aa_07a45d76ce34.slice/crio-40bd2c920c2965a27686999e333d746fba2c58610eb51497ba745e66ff199d99 WatchSource:0}: Error finding container 40bd2c920c2965a27686999e333d746fba2c58610eb51497ba745e66ff199d99: Status 404 returned error can't find the container with id 40bd2c920c2965a27686999e333d746fba2c58610eb51497ba745e66ff199d99 Nov 03 00:56:44.664592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:44.664567 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46"] Nov 03 00:56:45.017915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:45.017878 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" event={"ID":"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34","Type":"ContainerStarted","Data":"40bd2c920c2965a27686999e333d746fba2c58610eb51497ba745e66ff199d99"} Nov 03 00:56:46.029028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:46.028992 5144 generic.go:356] "Generic (PLEG): container finished" podID="6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" containerID="0992f39398c21011a0dfcc6ef5d5963669828084399b7142309bacb570faab68" exitCode=0 Nov 03 00:56:46.029445 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:46.029096 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" event={"ID":"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34","Type":"ContainerDied","Data":"0992f39398c21011a0dfcc6ef5d5963669828084399b7142309bacb570faab68"} Nov 03 00:56:47.202515 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.202491 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:47.271068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.271037 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh\" (UniqueName: \"kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh\") pod \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " Nov 03 00:56:47.271239 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.271202 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca\") pod \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " Nov 03 00:56:47.271357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.271322 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca\") pod \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\" (UID: \"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34\") " Nov 03 00:56:47.271547 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.271524 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca" pod "6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" (UID: "6a5bdc82-0cc5-4278-b4aa-07a45d76ce34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:47.271768 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.271749 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:47.273371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.273345 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh" pod "6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" (UID: "6a5bdc82-0cc5-4278-b4aa-07a45d76ce34"). InnerVolumeSpecName "kube-api-access-hc7jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:47.273478 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.273387 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca" pod "6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" (UID: "6a5bdc82-0cc5-4278-b4aa-07a45d76ce34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:47.373135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.373064 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:47.373135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:47.373092 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc7jh\" (UniqueName: \"kubernetes.io/projected/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34-kube-api-access-hc7jh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:48.049376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.049346 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" Nov 03 00:56:48.049752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.049375 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46" event={"ID":"6a5bdc82-0cc5-4278-b4aa-07a45d76ce34","Type":"ContainerDied","Data":"40bd2c920c2965a27686999e333d746fba2c58610eb51497ba745e66ff199d99"} Nov 03 00:56:48.049752 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.049437 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40bd2c920c2965a27686999e333d746fba2c58610eb51497ba745e66ff199d99" Nov 03 00:56:48.194149 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.194115 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q"] Nov 03 00:56:48.194859 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.194844 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" containerName="revoke-keys" Nov 03 00:56:48.194924 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.194861 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" containerName="revoke-keys" Nov 03 00:56:48.194961 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.194956 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" containerName="revoke-keys" Nov 03 00:56:48.198684 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.198665 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.253798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.253763 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q"] Nov 03 00:56:48.254261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.253881 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:56:48.254261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.253959 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:56:48.283984 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.283956 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pll2j\" (UniqueName: \"kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.284196 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.284173 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.284249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.284215 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.385300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.385258 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.385300 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.385304 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.385564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.385398 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pll2j\" (UniqueName: \"kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.386119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.386044 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.387917 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.387891 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.441545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.441512 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pll2j\" (UniqueName: \"kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j\") pod \"maas-api-revoke-sub-836ef7f5-7dp4q\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.512082 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.512020 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:48.898930 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:48.898900 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q"] Nov 03 00:56:48.900523 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:48.900493 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbffd7e9_cc44_4ee1_b6fa_4693fae3b501.slice/crio-e7482bf1eed8e78e72d664394be5e0dfb2115c25d92f4978caed7b27e28e7d66 WatchSource:0}: Error finding container e7482bf1eed8e78e72d664394be5e0dfb2115c25d92f4978caed7b27e28e7d66: Status 404 returned error can't find the container with id e7482bf1eed8e78e72d664394be5e0dfb2115c25d92f4978caed7b27e28e7d66 Nov 03 00:56:49.067710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:49.067674 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" event={"ID":"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501","Type":"ContainerStarted","Data":"e7482bf1eed8e78e72d664394be5e0dfb2115c25d92f4978caed7b27e28e7d66"} Nov 03 00:56:50.076703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.076668 5144 generic.go:356] "Generic (PLEG): container finished" podID="cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" containerID="9ecb222291d118a599c6bf29f59f223140d6d4943b33d09b46f7c49f7e998032" exitCode=0 Nov 03 00:56:50.077090 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.076736 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" event={"ID":"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501","Type":"ContainerDied","Data":"9ecb222291d118a599c6bf29f59f223140d6d4943b33d09b46f7c49f7e998032"} Nov 03 00:56:50.430039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.430007 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb"] Nov 03 00:56:50.434499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.434476 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.486331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.486291 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c19567\"" Nov 03 00:56:50.486495 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.486462 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c19567-dockercfg-9ddg4\"" Nov 03 00:56:50.489603 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.489578 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb"] Nov 03 00:56:50.560845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.528043 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4d4bf\" (UniqueName: \"kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf\") pod \"payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.566135 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.566101 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.586469 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.586438 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8"] Nov 03 00:56:50.591823 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.591796 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:50.647324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.647293 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8"] Nov 03 00:56:50.667484 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.667440 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.668137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.667609 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4d4bf\" (UniqueName: \"kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf\") pod \"payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.668322 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.668302 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.724526 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.724459 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4d4bf\" (UniqueName: \"kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf\") pod \"payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.749563 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.749535 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:50.768550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.768517 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66d5r\" (UniqueName: \"kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r\") pod \"payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:50.768796 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.768777 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:50.870494 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.870457 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66d5r\" (UniqueName: \"kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r\") pod \"payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:50.870675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.870650 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:50.871313 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.871287 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:50.927565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:50.927533 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-66d5r\" (UniqueName: \"kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r\") pod \"payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:51.130824 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.130793 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb"] Nov 03 00:56:51.131572 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:51.131545 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e78b96a_9903_4fd9_80d1_752cdd022a17.slice/crio-9d4c5965d848d66e0536714447e4da1a29d64a27a255f1144e4bc423642c7995 WatchSource:0}: Error finding container 9d4c5965d848d66e0536714447e4da1a29d64a27a255f1144e4bc423642c7995: Status 404 returned error can't find the container with id 9d4c5965d848d66e0536714447e4da1a29d64a27a255f1144e4bc423642c7995 Nov 03 00:56:51.207529 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.207502 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:51.235596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.235571 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:51.379607 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.379520 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca\") pod \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " Nov 03 00:56:51.379798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.379689 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca\") pod \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " Nov 03 00:56:51.379798 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.379734 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j\" (UniqueName: \"kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j\") pod \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\" (UID: \"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501\") " Nov 03 00:56:51.380016 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.379986 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca" pod "cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" (UID: "cbffd7e9-cc44-4ee1-b6fa-4693fae3b501"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:51.380565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.380376 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:51.381993 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.381967 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca" pod "cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" (UID: "cbffd7e9-cc44-4ee1-b6fa-4693fae3b501"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:51.382533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.382512 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j" pod "cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" (UID: "cbffd7e9-cc44-4ee1-b6fa-4693fae3b501"). InnerVolumeSpecName "kube-api-access-pll2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:51.481597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.481564 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:51.481597 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.481595 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pll2j\" (UniqueName: \"kubernetes.io/projected/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501-kube-api-access-pll2j\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:51.588472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:51.588437 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8"] Nov 03 00:56:51.589349 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:51.589314 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d9c691_6f27_40d2_be43_3b1ecd24526c.slice/crio-97eeedbddfec74ae49776f296126d7106c40d5d82ff85367ca8bbed3ad6c1e9a WatchSource:0}: Error finding container 97eeedbddfec74ae49776f296126d7106c40d5d82ff85367ca8bbed3ad6c1e9a: Status 404 returned error can't find the container with id 97eeedbddfec74ae49776f296126d7106c40d5d82ff85367ca8bbed3ad6c1e9a Nov 03 00:56:52.094079 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.094044 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" event={"ID":"38d9c691-6f27-40d2-be43-3b1ecd24526c","Type":"ContainerStarted","Data":"f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453"} Nov 03 00:56:52.094288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.094129 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" event={"ID":"38d9c691-6f27-40d2-be43-3b1ecd24526c","Type":"ContainerStarted","Data":"97eeedbddfec74ae49776f296126d7106c40d5d82ff85367ca8bbed3ad6c1e9a"} Nov 03 00:56:52.094288 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.094159 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:56:52.095848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.095821 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" event={"ID":"9e78b96a-9903-4fd9-80d1-752cdd022a17","Type":"ContainerStarted","Data":"0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede"} Nov 03 00:56:52.095981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.095872 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" event={"ID":"9e78b96a-9903-4fd9-80d1-752cdd022a17","Type":"ContainerStarted","Data":"9d4c5965d848d66e0536714447e4da1a29d64a27a255f1144e4bc423642c7995"} Nov 03 00:56:52.095981 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.095915 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:56:52.097145 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.097128 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" Nov 03 00:56:52.097237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.097161 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q" event={"ID":"cbffd7e9-cc44-4ee1-b6fa-4693fae3b501","Type":"ContainerDied","Data":"e7482bf1eed8e78e72d664394be5e0dfb2115c25d92f4978caed7b27e28e7d66"} Nov 03 00:56:52.097237 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.097195 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7482bf1eed8e78e72d664394be5e0dfb2115c25d92f4978caed7b27e28e7d66" Nov 03 00:56:52.209178 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.209130 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" podStartSLOduration=2.209114061 podStartE2EDuration="2.209114061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:52.206036572 +0000 UTC m=+3102.351731028" watchObservedRunningTime="2026-11-03 00:56:52.209114061 +0000 UTC m=+3102.354808517" Nov 03 00:56:52.324006 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:52.323864 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" podStartSLOduration=2.3238503489999998 podStartE2EDuration="2.323850349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:52.31978165 +0000 UTC m=+3102.465476132" watchObservedRunningTime="2026-11-03 00:56:52.323850349 +0000 UTC m=+3102.469544834" Nov 03 00:56:54.690889 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.690849 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr"] Nov 03 00:56:54.691912 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.691895 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" containerName="revoke-keys" Nov 03 00:56:54.691976 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.691915 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" containerName="revoke-keys" Nov 03 00:56:54.692094 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.692083 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" containerName="revoke-keys" Nov 03 00:56:54.696061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.696037 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.721349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.721314 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.721500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.721459 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.721583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.721513 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6bt8\" (UniqueName: \"kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.749589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.749562 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr"] Nov 03 00:56:54.794023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.793995 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 00:56:54.794023 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.794015 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 00:56:54.823285 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.823247 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.823472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.823339 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.823472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.823374 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6bt8\" (UniqueName: \"kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.824005 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.823984 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.825857 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.825834 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:54.879839 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:54.879811 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6bt8\" (UniqueName: \"kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8\") pod \"maas-api-revoke-sub-441ee790-6q5wr\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:55.013997 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:55.013965 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:55.391795 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:55.391764 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf6c5d92_02ed_45ec_894d_82ce24703fe1.slice/crio-1d49684e080fecf977e22a2550ef6b13b890c60fb229e6567dc516c63045468c WatchSource:0}: Error finding container 1d49684e080fecf977e22a2550ef6b13b890c60fb229e6567dc516c63045468c: Status 404 returned error can't find the container with id 1d49684e080fecf977e22a2550ef6b13b890c60fb229e6567dc516c63045468c Nov 03 00:56:55.392520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:55.392496 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr"] Nov 03 00:56:56.136049 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:56.136012 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" event={"ID":"bf6c5d92-02ed-45ec-894d-82ce24703fe1","Type":"ContainerStarted","Data":"a245a5da5bb2908b18ac59e479bcdc5edfeb2ccf3d995424e4b7c18daab0f0f8"} Nov 03 00:56:56.136550 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:56.136069 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" event={"ID":"bf6c5d92-02ed-45ec-894d-82ce24703fe1","Type":"ContainerStarted","Data":"1d49684e080fecf977e22a2550ef6b13b890c60fb229e6567dc516c63045468c"} Nov 03 00:56:56.250115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:56.250062 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" podStartSLOduration=2.250048686 podStartE2EDuration="2.250048686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:56.247965477 +0000 UTC m=+3106.393659952" watchObservedRunningTime="2026-11-03 00:56:56.250048686 +0000 UTC m=+3106.395743141" Nov 03 00:56:57.146289 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:57.146253 5144 generic.go:356] "Generic (PLEG): container finished" podID="bf6c5d92-02ed-45ec-894d-82ce24703fe1" containerID="a245a5da5bb2908b18ac59e479bcdc5edfeb2ccf3d995424e4b7c18daab0f0f8" exitCode=0 Nov 03 00:56:57.146737 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:57.146342 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" event={"ID":"bf6c5d92-02ed-45ec-894d-82ce24703fe1","Type":"ContainerDied","Data":"a245a5da5bb2908b18ac59e479bcdc5edfeb2ccf3d995424e4b7c18daab0f0f8"} Nov 03 00:56:58.307666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.307615 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:58.368742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.368711 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8\" (UniqueName: \"kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8\") pod \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " Nov 03 00:56:58.368898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.368750 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca\") pod \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " Nov 03 00:56:58.368898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.368811 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca\") pod \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\" (UID: \"bf6c5d92-02ed-45ec-894d-82ce24703fe1\") " Nov 03 00:56:58.369215 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.369193 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca" pod "bf6c5d92-02ed-45ec-894d-82ce24703fe1" (UID: "bf6c5d92-02ed-45ec-894d-82ce24703fe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:56:58.369381 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.369363 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:58.371024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.370992 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca" pod "bf6c5d92-02ed-45ec-894d-82ce24703fe1" (UID: "bf6c5d92-02ed-45ec-894d-82ce24703fe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:56:58.371127 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.371108 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8" pod "bf6c5d92-02ed-45ec-894d-82ce24703fe1" (UID: "bf6c5d92-02ed-45ec-894d-82ce24703fe1"). InnerVolumeSpecName "kube-api-access-l6bt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:56:58.470849 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.470756 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6bt8\" (UniqueName: \"kubernetes.io/projected/bf6c5d92-02ed-45ec-894d-82ce24703fe1-kube-api-access-l6bt8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:58.470849 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.470790 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf6c5d92-02ed-45ec-894d-82ce24703fe1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:56:58.936364 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.936330 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2"] Nov 03 00:56:58.937057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.937038 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bf6c5d92-02ed-45ec-894d-82ce24703fe1" containerName="revoke-keys" Nov 03 00:56:58.937057 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.937054 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf6c5d92-02ed-45ec-894d-82ce24703fe1" containerName="revoke-keys" Nov 03 00:56:58.937230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.937163 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf6c5d92-02ed-45ec-894d-82ce24703fe1" containerName="revoke-keys" Nov 03 00:56:58.940836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.940811 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:58.977209 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.977180 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:58.977390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.977323 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptf75\" (UniqueName: \"kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:58.977453 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.977421 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:58.996278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:58.996248 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2"] Nov 03 00:56:59.079197 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.079161 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptf75\" (UniqueName: \"kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.079357 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.079255 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.079434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.079361 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.080047 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.080028 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.081920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.081897 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.136071 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.136045 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptf75\" (UniqueName: \"kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75\") pod \"maas-api-revoke-sub-f7b976a3-mpcc2\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.164182 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.164155 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" Nov 03 00:56:59.164373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.164181 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr" event={"ID":"bf6c5d92-02ed-45ec-894d-82ce24703fe1","Type":"ContainerDied","Data":"1d49684e080fecf977e22a2550ef6b13b890c60fb229e6567dc516c63045468c"} Nov 03 00:56:59.164373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.164225 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d49684e080fecf977e22a2550ef6b13b890c60fb229e6567dc516c63045468c" Nov 03 00:56:59.256849 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.256813 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:56:59.648373 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:56:59.648346 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2"] Nov 03 00:56:59.649333 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:56:59.649313 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20315322_e2ce_46a6_96c7_f00ca1304cc3.slice/crio-e11d1b7e890ac4ee4ebeaf444349f5e4fdd577953fc2f1ccd5e29fc2e71b3da3 WatchSource:0}: Error finding container e11d1b7e890ac4ee4ebeaf444349f5e4fdd577953fc2f1ccd5e29fc2e71b3da3: Status 404 returned error can't find the container with id e11d1b7e890ac4ee4ebeaf444349f5e4fdd577953fc2f1ccd5e29fc2e71b3da3 Nov 03 00:57:00.185299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:00.185210 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" event={"ID":"20315322-e2ce-46a6-96c7-f00ca1304cc3","Type":"ContainerStarted","Data":"6a7f57d698e2d73b74eaa01389328052a513b4f7e73626651e4b9c3e6db4425d"} Nov 03 00:57:00.185299 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:00.185288 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" event={"ID":"20315322-e2ce-46a6-96c7-f00ca1304cc3","Type":"ContainerStarted","Data":"e11d1b7e890ac4ee4ebeaf444349f5e4fdd577953fc2f1ccd5e29fc2e71b3da3"} Nov 03 00:57:00.302230 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:00.302173 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" podStartSLOduration=2.30216002 podStartE2EDuration="2.30216002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:00.295947503 +0000 UTC m=+3110.441641962" watchObservedRunningTime="2026-11-03 00:57:00.30216002 +0000 UTC m=+3110.447854475" Nov 03 00:57:01.195233 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:01.195196 5144 generic.go:356] "Generic (PLEG): container finished" podID="20315322-e2ce-46a6-96c7-f00ca1304cc3" containerID="6a7f57d698e2d73b74eaa01389328052a513b4f7e73626651e4b9c3e6db4425d" exitCode=0 Nov 03 00:57:01.195691 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:01.195282 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" event={"ID":"20315322-e2ce-46a6-96c7-f00ca1304cc3","Type":"ContainerDied","Data":"6a7f57d698e2d73b74eaa01389328052a513b4f7e73626651e4b9c3e6db4425d"} Nov 03 00:57:02.386220 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.386197 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:57:02.532207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.532165 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca\") pod \"20315322-e2ce-46a6-96c7-f00ca1304cc3\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " Nov 03 00:57:02.532397 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.532270 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca\") pod \"20315322-e2ce-46a6-96c7-f00ca1304cc3\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " Nov 03 00:57:02.532440 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.532416 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75\" (UniqueName: \"kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75\") pod \"20315322-e2ce-46a6-96c7-f00ca1304cc3\" (UID: \"20315322-e2ce-46a6-96c7-f00ca1304cc3\") " Nov 03 00:57:02.532604 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.532578 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca" pod "20315322-e2ce-46a6-96c7-f00ca1304cc3" (UID: "20315322-e2ce-46a6-96c7-f00ca1304cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:57:02.532944 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.532922 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:02.535612 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.535580 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75" pod "20315322-e2ce-46a6-96c7-f00ca1304cc3" (UID: "20315322-e2ce-46a6-96c7-f00ca1304cc3"). InnerVolumeSpecName "kube-api-access-ptf75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:02.537739 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.537710 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca" pod "20315322-e2ce-46a6-96c7-f00ca1304cc3" (UID: "20315322-e2ce-46a6-96c7-f00ca1304cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:02.634583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.634544 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptf75\" (UniqueName: \"kubernetes.io/projected/20315322-e2ce-46a6-96c7-f00ca1304cc3-kube-api-access-ptf75\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:02.634583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:02.634580 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20315322-e2ce-46a6-96c7-f00ca1304cc3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:03.111825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:03.111789 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:57:03.112030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:03.111855 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:57:03.211648 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:03.211606 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" Nov 03 00:57:03.211848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:03.211650 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2" event={"ID":"20315322-e2ce-46a6-96c7-f00ca1304cc3","Type":"ContainerDied","Data":"e11d1b7e890ac4ee4ebeaf444349f5e4fdd577953fc2f1ccd5e29fc2e71b3da3"} Nov 03 00:57:03.211848 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:03.211699 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e11d1b7e890ac4ee4ebeaf444349f5e4fdd577953fc2f1ccd5e29fc2e71b3da3" Nov 03 00:57:12.156552 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:57:12.156512 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:57:14.422970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:14.422939 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9"] Nov 03 00:57:29.444122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.444078 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" podUID="5c79a872-0552-41d7-8bb2-686de23704aa" containerName="main" containerID="cri-o://2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95" gracePeriod=45 Nov 03 00:57:29.727748 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.727718 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:57:29.911808 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.911774 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs\") pod \"5c79a872-0552-41d7-8bb2-686de23704aa\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " Nov 03 00:57:29.911987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.911874 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm\") pod \"5c79a872-0552-41d7-8bb2-686de23704aa\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " Nov 03 00:57:29.911987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.911917 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home\") pod \"5c79a872-0552-41d7-8bb2-686de23704aa\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " Nov 03 00:57:29.911987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.911959 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir\") pod \"5c79a872-0552-41d7-8bb2-686de23704aa\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " Nov 03 00:57:29.912119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912051 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache\") pod \"5c79a872-0552-41d7-8bb2-686de23704aa\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " Nov 03 00:57:29.912119 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912085 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv\" (UniqueName: \"kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv\") pod \"5c79a872-0552-41d7-8bb2-686de23704aa\" (UID: \"5c79a872-0552-41d7-8bb2-686de23704aa\") " Nov 03 00:57:29.912263 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912208 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home" pod "5c79a872-0552-41d7-8bb2-686de23704aa" (UID: "5c79a872-0552-41d7-8bb2-686de23704aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:29.912339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912283 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir" pod "5c79a872-0552-41d7-8bb2-686de23704aa" (UID: "5c79a872-0552-41d7-8bb2-686de23704aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:29.912339 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912291 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache" pod "5c79a872-0552-41d7-8bb2-686de23704aa" (UID: "5c79a872-0552-41d7-8bb2-686de23704aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:29.912647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912607 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:29.912647 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912649 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:29.912826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.912663 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:29.914432 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.914405 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm" pod "5c79a872-0552-41d7-8bb2-686de23704aa" (UID: "5c79a872-0552-41d7-8bb2-686de23704aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:29.914533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.914425 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs" pod "5c79a872-0552-41d7-8bb2-686de23704aa" (UID: "5c79a872-0552-41d7-8bb2-686de23704aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:29.914533 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:29.914438 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv" pod "5c79a872-0552-41d7-8bb2-686de23704aa" (UID: "5c79a872-0552-41d7-8bb2-686de23704aa"). InnerVolumeSpecName "kube-api-access-9w4sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:30.014005 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.013976 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c79a872-0552-41d7-8bb2-686de23704aa-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:30.014137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.014007 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w4sv\" (UniqueName: \"kubernetes.io/projected/5c79a872-0552-41d7-8bb2-686de23704aa-kube-api-access-9w4sv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:30.014137 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.014023 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c79a872-0552-41d7-8bb2-686de23704aa-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:30.429030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.428928 5144 generic.go:356] "Generic (PLEG): container finished" podID="5c79a872-0552-41d7-8bb2-686de23704aa" containerID="2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95" exitCode=0 Nov 03 00:57:30.429030 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.428971 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" Nov 03 00:57:30.429232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.429021 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" event={"ID":"5c79a872-0552-41d7-8bb2-686de23704aa","Type":"ContainerDied","Data":"2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95"} Nov 03 00:57:30.429232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.429068 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9" event={"ID":"5c79a872-0552-41d7-8bb2-686de23704aa","Type":"ContainerDied","Data":"84097f8a2886233f734971514589536088ce91a68379fc759fd3b9ea8e26e334"} Nov 03 00:57:30.429232 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.429085 5144 scope.go:122] "RemoveContainer" containerID="2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95" Nov 03 00:57:30.439923 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.439896 5144 scope.go:122] "RemoveContainer" containerID="2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95" Nov 03 00:57:30.440187 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:57:30.440166 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95\": container with ID starting with 2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95 not found: ID does not exist" containerID="2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95" Nov 03 00:57:30.440249 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.440199 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95"} err="failed to get container status \"2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95\": rpc error: code = NotFound desc = could not find container \"2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95\": container with ID starting with 2e33f6aca87ad5ca68e39b1dc694b2068f80b12a4dad42664299d1885cd14d95 not found: ID does not exist" Nov 03 00:57:30.592803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.592776 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9"] Nov 03 00:57:30.642570 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:30.642535 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-fce94b/auth-test-model-fce94b-kserve-6fd858f87c-tjcx9"] Nov 03 00:57:31.998506 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:31.998473 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c79a872-0552-41d7-8bb2-686de23704aa" path="/var/lib/kubelet/pods/5c79a872-0552-41d7-8bb2-686de23704aa/volumes" Nov 03 00:57:32.650363 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:32.650325 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq"] Nov 03 00:57:42.155878 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:57:42.155836 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:57:43.359909 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:43.359876 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft"] Nov 03 00:57:45.932248 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.932210 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4"] Nov 03 00:57:45.933234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.933215 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5c79a872-0552-41d7-8bb2-686de23704aa" containerName="main" Nov 03 00:57:45.933234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.933233 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c79a872-0552-41d7-8bb2-686de23704aa" containerName="main" Nov 03 00:57:45.933403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.933258 5144 cpu_manager.go:414] "containerMap: removing container" podUID="20315322-e2ce-46a6-96c7-f00ca1304cc3" containerName="revoke-keys" Nov 03 00:57:45.933403 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.933266 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20315322-e2ce-46a6-96c7-f00ca1304cc3" containerName="revoke-keys" Nov 03 00:57:45.933512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.933404 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c79a872-0552-41d7-8bb2-686de23704aa" containerName="main" Nov 03 00:57:45.933512 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.933417 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="20315322-e2ce-46a6-96c7-f00ca1304cc3" containerName="revoke-keys" Nov 03 00:57:45.953582 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:45.953549 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.008303 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.008275 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-ea50b7-openshift-default-dockercfg-9hwl8\"" Nov 03 00:57:46.019881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.019846 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.021690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.021659 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5d8k\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.021956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.021940 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.022097 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.022081 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.022291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.022272 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.022444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.022428 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.022564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.022548 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.022872 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.022854 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.023028 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.023011 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.062777 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.062743 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4"] Nov 03 00:57:46.124561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.124522 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.124561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.124558 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.124934 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.124907 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125056 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125040 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125087 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125151 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125246 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125182 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5d8k\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125342 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125317 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125325 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125465 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125446 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125546 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125041 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.125888 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125865 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.126003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.125978 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.128012 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.127990 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.128376 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.128354 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.182476 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.182398 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.182666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.182489 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5d8k\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k\") pod \"e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.272084 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.272051 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:46.717648 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:57:46.717600 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddceed9b_f049_40aa_9db7_246a770583d4.slice/crio-17680345732cdd34a50197b3498f94e3d3f8a473fd29dca5187b0dbbefb10969 WatchSource:0}: Error finding container 17680345732cdd34a50197b3498f94e3d3f8a473fd29dca5187b0dbbefb10969: Status 404 returned error can't find the container with id 17680345732cdd34a50197b3498f94e3d3f8a473fd29dca5187b0dbbefb10969 Nov 03 00:57:46.717826 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.717701 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4"] Nov 03 00:57:46.720846 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.720817 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:57:46.720943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.720877 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:57:46.720943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:46.720906 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:57:47.589690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:47.588509 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" event={"ID":"ddceed9b-f049-40aa-9db7-246a770583d4","Type":"ContainerStarted","Data":"e8239d2dafdfaf06e919bcad0188df012baac4ef850163254ded98533a676106"} Nov 03 00:57:47.589690 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:47.588572 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" event={"ID":"ddceed9b-f049-40aa-9db7-246a770583d4","Type":"ContainerStarted","Data":"17680345732cdd34a50197b3498f94e3d3f8a473fd29dca5187b0dbbefb10969"} Nov 03 00:57:47.671623 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:47.671574 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" podUID="076dff53-bf61-4a5b-8789-8d6342df3a70" containerName="main" containerID="cri-o://758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5" gracePeriod=45 Nov 03 00:57:47.762661 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:47.762516 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podStartSLOduration=2.762496689 podStartE2EDuration="2.762496689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:47.75418565 +0000 UTC m=+3157.899880124" watchObservedRunningTime="2026-11-03 00:57:47.762496689 +0000 UTC m=+3157.908191146" Nov 03 00:57:47.778727 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:47.777576 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" podUID="076dff53-bf61-4a5b-8789-8d6342df3a70" containerName="main" probeResult="failure" output="Get \"https://10.132.0.251:8000/ready\": dial tcp 10.132.0.251:8000: connect: connection refused" Nov 03 00:57:48.002378 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.002347 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:57:48.153219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153136 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache\") pod \"076dff53-bf61-4a5b-8789-8d6342df3a70\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " Nov 03 00:57:48.153219 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153204 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs\" (UniqueName: \"kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs\") pod \"076dff53-bf61-4a5b-8789-8d6342df3a70\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " Nov 03 00:57:48.153472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153261 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5\" (UniqueName: \"kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5\") pod \"076dff53-bf61-4a5b-8789-8d6342df3a70\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " Nov 03 00:57:48.153472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153317 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home\") pod \"076dff53-bf61-4a5b-8789-8d6342df3a70\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " Nov 03 00:57:48.153472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153353 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm\") pod \"076dff53-bf61-4a5b-8789-8d6342df3a70\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " Nov 03 00:57:48.153472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153414 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache" pod "076dff53-bf61-4a5b-8789-8d6342df3a70" (UID: "076dff53-bf61-4a5b-8789-8d6342df3a70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:48.153472 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153446 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir\") pod \"076dff53-bf61-4a5b-8789-8d6342df3a70\" (UID: \"076dff53-bf61-4a5b-8789-8d6342df3a70\") " Nov 03 00:57:48.153803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153533 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home" pod "076dff53-bf61-4a5b-8789-8d6342df3a70" (UID: "076dff53-bf61-4a5b-8789-8d6342df3a70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:48.153803 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.153760 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir" pod "076dff53-bf61-4a5b-8789-8d6342df3a70" (UID: "076dff53-bf61-4a5b-8789-8d6342df3a70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:48.154181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.154111 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:48.154181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.154131 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:48.154181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.154139 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:48.155741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.155719 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm" pod "076dff53-bf61-4a5b-8789-8d6342df3a70" (UID: "076dff53-bf61-4a5b-8789-8d6342df3a70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:48.155741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.155728 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs" pod "076dff53-bf61-4a5b-8789-8d6342df3a70" (UID: "076dff53-bf61-4a5b-8789-8d6342df3a70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:48.155963 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.155945 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5" pod "076dff53-bf61-4a5b-8789-8d6342df3a70" (UID: "076dff53-bf61-4a5b-8789-8d6342df3a70"). InnerVolumeSpecName "kube-api-access-whsb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:48.255651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.255603 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whsb5\" (UniqueName: \"kubernetes.io/projected/076dff53-bf61-4a5b-8789-8d6342df3a70-kube-api-access-whsb5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:48.255821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.255665 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/076dff53-bf61-4a5b-8789-8d6342df3a70-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:48.255821 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.255677 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/076dff53-bf61-4a5b-8789-8d6342df3a70-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:48.273225 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.273193 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:48.280226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.280200 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:48.597343 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.597307 5144 generic.go:356] "Generic (PLEG): container finished" podID="076dff53-bf61-4a5b-8789-8d6342df3a70" containerID="758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5" exitCode=0 Nov 03 00:57:48.597898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.597360 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" Nov 03 00:57:48.597898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.597391 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" event={"ID":"076dff53-bf61-4a5b-8789-8d6342df3a70","Type":"ContainerDied","Data":"758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5"} Nov 03 00:57:48.597898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.597440 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq" event={"ID":"076dff53-bf61-4a5b-8789-8d6342df3a70","Type":"ContainerDied","Data":"ec2c1e4747e898d7266a6e6357a30fba1b9b307ab88ba14e2721529e95f8e4ed"} Nov 03 00:57:48.597898 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.597462 5144 scope.go:122] "RemoveContainer" containerID="758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5" Nov 03 00:57:48.598152 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.597909 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:48.599067 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.599047 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:57:48.614900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.614882 5144 scope.go:122] "RemoveContainer" containerID="758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5" Nov 03 00:57:48.615161 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:57:48.615145 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5\": container with ID starting with 758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5 not found: ID does not exist" containerID="758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5" Nov 03 00:57:48.615224 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.615169 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5"} err="failed to get container status \"758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5\": rpc error: code = NotFound desc = could not find container \"758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5\": container with ID starting with 758a85d9aff17855ddc7f3cee76deddfc68e55e25e31d9bedaed1c0e945587f5 not found: ID does not exist" Nov 03 00:57:48.762261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.762228 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq"] Nov 03 00:57:48.765959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:48.765937 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-e4c8b1/auth-test-model-e4c8b1-kserve-ff9f56677-p4hdq"] Nov 03 00:57:49.999356 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:49.999321 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="076dff53-bf61-4a5b-8789-8d6342df3a70" path="/var/lib/kubelet/pods/076dff53-bf61-4a5b-8789-8d6342df3a70/volumes" Nov 03 00:57:50.359764 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.359728 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64"] Nov 03 00:57:50.361121 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.361091 5144 cpu_manager.go:414] "containerMap: removing container" podUID="076dff53-bf61-4a5b-8789-8d6342df3a70" containerName="main" Nov 03 00:57:50.370746 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.370716 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="076dff53-bf61-4a5b-8789-8d6342df3a70" containerName="main" Nov 03 00:57:50.371003 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.370990 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="076dff53-bf61-4a5b-8789-8d6342df3a70" containerName="main" Nov 03 00:57:50.376122 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.376093 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.403455 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.403431 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64"] Nov 03 00:57:50.448068 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.448035 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-3b1d36\"" Nov 03 00:57:50.448333 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.448310 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-3b1d36-dockercfg-jbq8z\"" Nov 03 00:57:50.492710 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.492684 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.492836 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.492818 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rq57\" (UniqueName: \"kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57\") pod \"payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.594365 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.594317 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rq57\" (UniqueName: \"kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57\") pod \"payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.594579 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.594559 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.595241 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.595216 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.650741 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.650673 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rq57\" (UniqueName: \"kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57\") pod \"payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:50.694920 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:50.694887 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:51.074023 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:57:51.073994 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12aa25ab_ade3_4157_a88a_74b1ed30ee8d.slice/crio-f4f4719c9085fbbb740af23726ab4edc969189e509419a78ad08a02977685c06 WatchSource:0}: Error finding container f4f4719c9085fbbb740af23726ab4edc969189e509419a78ad08a02977685c06: Status 404 returned error can't find the container with id f4f4719c9085fbbb740af23726ab4edc969189e509419a78ad08a02977685c06 Nov 03 00:57:51.074499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:51.074476 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64"] Nov 03 00:57:51.623960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:51.623926 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" event={"ID":"12aa25ab-ade3-4157-a88a-74b1ed30ee8d","Type":"ContainerStarted","Data":"7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad"} Nov 03 00:57:51.624117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:51.623977 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" event={"ID":"12aa25ab-ade3-4157-a88a-74b1ed30ee8d","Type":"ContainerStarted","Data":"f4f4719c9085fbbb740af23726ab4edc969189e509419a78ad08a02977685c06"} Nov 03 00:57:51.624186 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:51.624170 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:57:51.738405 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:51.738344 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" podStartSLOduration=1.7383213149999999 podStartE2EDuration="1.738321315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:51.738305132 +0000 UTC m=+3161.883999606" watchObservedRunningTime="2026-11-03 00:57:51.738321315 +0000 UTC m=+3161.884015771" Nov 03 00:57:55.362651 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.362600 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j"] Nov 03 00:57:55.366919 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.366898 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.370987 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.370964 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srgcq\" (UniqueName: \"kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq\") pod \"payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.371115 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.371031 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.422765 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.422739 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j"] Nov 03 00:57:55.470970 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.470934 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-ea50b7\"" Nov 03 00:57:55.510163 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.472028 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.510500 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.472931 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.510729 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.473107 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-ea50b7-dockercfg-r6vbn\"" Nov 03 00:57:55.511304 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.511271 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srgcq\" (UniqueName: \"kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq\") pod \"payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.535226 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.535192 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx"] Nov 03 00:57:55.541144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.541116 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.568167 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.568138 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srgcq\" (UniqueName: \"kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq\") pod \"payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.586682 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.581775 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx"] Nov 03 00:57:55.613438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.613368 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6x9s\" (UniqueName: \"kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s\") pod \"payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.613589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.613484 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume\") pod \"payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.681032 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.681003 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:55.715534 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.715503 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume\") pod \"payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.715772 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.715751 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6x9s\" (UniqueName: \"kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s\") pod \"payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.716267 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.716233 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume\") pod \"payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.779192 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.778690 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6x9s\" (UniqueName: \"kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s\") pod \"payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:55.861719 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:55.861685 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:56.078581 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:57:56.078551 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5802c243_b71a_4217_9775_17e6c6c5008c.slice/crio-fd1a6ccfa4f6246e1073c5360b94e29094ae8425094e9f522e519925fd2df578 WatchSource:0}: Error finding container fd1a6ccfa4f6246e1073c5360b94e29094ae8425094e9f522e519925fd2df578: Status 404 returned error can't find the container with id fd1a6ccfa4f6246e1073c5360b94e29094ae8425094e9f522e519925fd2df578 Nov 03 00:57:56.078780 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.078641 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j"] Nov 03 00:57:56.245461 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:57:56.245435 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a7df8b4_9939_47a7_8bd9_0a041dfdf3df.slice/crio-79e591c1afaf89bce8059ed3e8f765f15f49c3152b3689e72d6f6119edbdd1ea WatchSource:0}: Error finding container 79e591c1afaf89bce8059ed3e8f765f15f49c3152b3689e72d6f6119edbdd1ea: Status 404 returned error can't find the container with id 79e591c1afaf89bce8059ed3e8f765f15f49c3152b3689e72d6f6119edbdd1ea Nov 03 00:57:56.246409 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.246379 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx"] Nov 03 00:57:56.668521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.668492 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" event={"ID":"5802c243-b71a-4217-9775-17e6c6c5008c","Type":"ContainerStarted","Data":"b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55"} Nov 03 00:57:56.669017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.668548 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" event={"ID":"5802c243-b71a-4217-9775-17e6c6c5008c","Type":"ContainerStarted","Data":"fd1a6ccfa4f6246e1073c5360b94e29094ae8425094e9f522e519925fd2df578"} Nov 03 00:57:56.669017 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.668577 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:57:56.670147 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.670121 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" event={"ID":"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df","Type":"ContainerStarted","Data":"b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d"} Nov 03 00:57:56.670278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.670167 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" event={"ID":"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df","Type":"ContainerStarted","Data":"79e591c1afaf89bce8059ed3e8f765f15f49c3152b3689e72d6f6119edbdd1ea"} Nov 03 00:57:56.670278 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.670248 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:57:56.783545 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.783498 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" podStartSLOduration=1.783485123 podStartE2EDuration="1.783485123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:56.78141809 +0000 UTC m=+3166.927112547" watchObservedRunningTime="2026-11-03 00:57:56.783485123 +0000 UTC m=+3166.929179611" Nov 03 00:57:56.907669 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:56.899359 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" podStartSLOduration=1.899333852 podStartE2EDuration="1.899333852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:56.896250257 +0000 UTC m=+3167.041944714" watchObservedRunningTime="2026-11-03 00:57:56.899333852 +0000 UTC m=+3167.045028309" Nov 03 00:57:58.381679 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.381621 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" podUID="397f9a92-52ab-41b0-9e59-391189e7e04e" containerName="main" containerID="cri-o://1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53" gracePeriod=45 Nov 03 00:57:58.680927 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.680899 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:57:58.686344 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.686313 5144 generic.go:356] "Generic (PLEG): container finished" podID="397f9a92-52ab-41b0-9e59-391189e7e04e" containerID="1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53" exitCode=0 Nov 03 00:57:58.686488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.686364 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" Nov 03 00:57:58.686488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.686401 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" event={"ID":"397f9a92-52ab-41b0-9e59-391189e7e04e","Type":"ContainerDied","Data":"1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53"} Nov 03 00:57:58.686488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.686453 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft" event={"ID":"397f9a92-52ab-41b0-9e59-391189e7e04e","Type":"ContainerDied","Data":"14034b5af441343073865a9a03d290e85df61a9feac555b83dfabb53bb509979"} Nov 03 00:57:58.686488 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.686475 5144 scope.go:122] "RemoveContainer" containerID="1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53" Nov 03 00:57:58.697613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.697593 5144 scope.go:122] "RemoveContainer" containerID="1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53" Nov 03 00:57:58.697903 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:57:58.697885 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53\": container with ID starting with 1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53 not found: ID does not exist" containerID="1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53" Nov 03 00:57:58.697959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.697912 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53"} err="failed to get container status \"1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53\": rpc error: code = NotFound desc = could not find container \"1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53\": container with ID starting with 1a1f53a4363faf14d86ca6bda989456eac19efc68d3f162c4cea2994bef0ad53 not found: ID does not exist" Nov 03 00:57:58.752720 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.752690 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache\") pod \"397f9a92-52ab-41b0-9e59-391189e7e04e\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " Nov 03 00:57:58.752886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.752743 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir\") pod \"397f9a92-52ab-41b0-9e59-391189e7e04e\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " Nov 03 00:57:58.752886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.752809 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home\") pod \"397f9a92-52ab-41b0-9e59-391189e7e04e\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " Nov 03 00:57:58.752886 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.752882 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm\") pod \"397f9a92-52ab-41b0-9e59-391189e7e04e\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " Nov 03 00:57:58.753048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.752964 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache" pod "397f9a92-52ab-41b0-9e59-391189e7e04e" (UID: "397f9a92-52ab-41b0-9e59-391189e7e04e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:58.753048 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753009 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home" pod "397f9a92-52ab-41b0-9e59-391189e7e04e" (UID: "397f9a92-52ab-41b0-9e59-391189e7e04e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:58.753157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753052 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir" pod "397f9a92-52ab-41b0-9e59-391189e7e04e" (UID: "397f9a92-52ab-41b0-9e59-391189e7e04e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:58.753157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753101 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs\" (UniqueName: \"kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs\") pod \"397f9a92-52ab-41b0-9e59-391189e7e04e\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " Nov 03 00:57:58.753157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753142 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql\" (UniqueName: \"kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql\") pod \"397f9a92-52ab-41b0-9e59-391189e7e04e\" (UID: \"397f9a92-52ab-41b0-9e59-391189e7e04e\") " Nov 03 00:57:58.753698 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753677 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:58.753819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753703 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:58.753819 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.753716 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:58.755460 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.755431 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm" pod "397f9a92-52ab-41b0-9e59-391189e7e04e" (UID: "397f9a92-52ab-41b0-9e59-391189e7e04e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:57:58.755596 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.755578 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql" pod "397f9a92-52ab-41b0-9e59-391189e7e04e" (UID: "397f9a92-52ab-41b0-9e59-391189e7e04e"). InnerVolumeSpecName "kube-api-access-8fpql". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:57:58.756112 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.756081 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs" pod "397f9a92-52ab-41b0-9e59-391189e7e04e" (UID: "397f9a92-52ab-41b0-9e59-391189e7e04e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:57:58.855234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.855197 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/397f9a92-52ab-41b0-9e59-391189e7e04e-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:58.855234 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.855235 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fpql\" (UniqueName: \"kubernetes.io/projected/397f9a92-52ab-41b0-9e59-391189e7e04e-kube-api-access-8fpql\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:58.855511 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:58.855249 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/397f9a92-52ab-41b0-9e59-391189e7e04e-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:57:59.107956 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:59.107918 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft"] Nov 03 00:57:59.111675 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:59.111644 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20f0a3/auth-test-model-20f0a3-kserve-66d967cd5-7c5ft"] Nov 03 00:57:59.997882 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:57:59.997848 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="397f9a92-52ab-41b0-9e59-391189e7e04e" path="/var/lib/kubelet/pods/397f9a92-52ab-41b0-9e59-391189e7e04e/volumes" Nov 03 00:58:02.634784 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:02.634754 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 00:58:05.418493 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.418461 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv"] Nov 03 00:58:05.419396 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.419374 5144 cpu_manager.go:414] "containerMap: removing container" podUID="397f9a92-52ab-41b0-9e59-391189e7e04e" containerName="main" Nov 03 00:58:05.419524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.419399 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="397f9a92-52ab-41b0-9e59-391189e7e04e" containerName="main" Nov 03 00:58:05.419617 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.419602 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="397f9a92-52ab-41b0-9e59-391189e7e04e" containerName="main" Nov 03 00:58:05.424738 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.424717 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.481960 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.481846 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv"] Nov 03 00:58:05.524734 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.524694 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-087f6c-openshift-default-dockercfg-qz9kn\"" Nov 03 00:58:05.544261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544234 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544269 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544314 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544491 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544418 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544581 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544519 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544677 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544609 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544723 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544694 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544805 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544790 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn7qz\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.544847 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.544820 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646472 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646570 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646610 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646767 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646830 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646881 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.646999 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn7qz\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.647045 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.647123 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.648444 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.647620 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.658045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.658006 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.658045 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.658018 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.658291 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.658106 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.658703 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.658682 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.659109 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.659080 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.660334 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.660310 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.719973 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.719893 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn7qz\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.721881 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.721856 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token\") pod \"e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:05.738265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:05.738239 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:06.176508 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.176471 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv"] Nov 03 00:58:06.176830 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:58:06.176784 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c6dc9be_ef7c_4aa7_aa6d_238dc7245803.slice/crio-47af7fc52e32b4326e72c175d26f49b658a85a5349ca639e1567ac20f1aa1085 WatchSource:0}: Error finding container 47af7fc52e32b4326e72c175d26f49b658a85a5349ca639e1567ac20f1aa1085: Status 404 returned error can't find the container with id 47af7fc52e32b4326e72c175d26f49b658a85a5349ca639e1567ac20f1aa1085 Nov 03 00:58:06.184583 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.184546 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:06.184693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.184647 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:06.184742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.184692 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:06.757520 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.757488 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" event={"ID":"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803","Type":"ContainerStarted","Data":"23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61"} Nov 03 00:58:06.757950 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.757542 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" event={"ID":"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803","Type":"ContainerStarted","Data":"47af7fc52e32b4326e72c175d26f49b658a85a5349ca639e1567ac20f1aa1085"} Nov 03 00:58:06.873810 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:06.873755 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podStartSLOduration=1.8737376239999999 podStartE2EDuration="1.873737624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:06.872706949 +0000 UTC m=+3177.018401426" watchObservedRunningTime="2026-11-03 00:58:06.873737624 +0000 UTC m=+3177.019432081" Nov 03 00:58:07.680845 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:07.680817 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 00:58:07.681061 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:07.680882 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 00:58:07.739173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:07.739139 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:07.746404 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:07.746376 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:07.766505 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:07.766478 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:07.767438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:07.767414 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 00:58:12.155991 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:58:12.155953 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:58:13.842825 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:13.842790 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7"] Nov 03 00:58:13.848501 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:13.848481 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:13.970900 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:13.970869 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-087f6c\"" Nov 03 00:58:13.974653 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:13.971231 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-087f6c-dockercfg-zxqdr\"" Nov 03 00:58:13.992735 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:13.977677 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7"] Nov 03 00:58:14.059438 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.059401 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrml\" (UniqueName: \"kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml\") pod \"payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.059666 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.059643 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.161516 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.161414 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrml\" (UniqueName: \"kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml\") pod \"payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.161702 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.161644 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.162310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.162289 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.218327 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.218291 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrml\" (UniqueName: \"kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml\") pod \"payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.462527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.462440 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:14.844565 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:14.844531 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7"] Nov 03 00:58:14.845665 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:58:14.845623 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9ed56a_6442_42c5_b222_576f0f0cf687.slice/crio-b7c182b86160cd9778be35719e99473508543c3ab138be485e9caac4c70d9223 WatchSource:0}: Error finding container b7c182b86160cd9778be35719e99473508543c3ab138be485e9caac4c70d9223: Status 404 returned error can't find the container with id b7c182b86160cd9778be35719e99473508543c3ab138be485e9caac4c70d9223 Nov 03 00:58:15.835223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:15.835183 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" event={"ID":"2c9ed56a-6442-42c5-b222-576f0f0cf687","Type":"ContainerStarted","Data":"a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844"} Nov 03 00:58:15.835410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:15.835251 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:15.835410 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:15.835271 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" event={"ID":"2c9ed56a-6442-42c5-b222-576f0f0cf687","Type":"ContainerStarted","Data":"b7c182b86160cd9778be35719e99473508543c3ab138be485e9caac4c70d9223"} Nov 03 00:58:15.954833 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:15.954755 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" podStartSLOduration=2.954736495 podStartE2EDuration="2.954736495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:15.948065574 +0000 UTC m=+3186.093760047" watchObservedRunningTime="2026-11-03 00:58:15.954736495 +0000 UTC m=+3186.100430952" Nov 03 00:58:26.847310 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:26.847281 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 00:58:31.783100 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:31.783060 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf"] Nov 03 00:58:36.561813 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:36.561769 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 00:58:36.562223 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:36.561856 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 00:58:38.039995 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:38.039958 5144 generic.go:356] "Generic (PLEG): container finished" podID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerID="8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b" exitCode=0 Nov 03 00:58:38.040434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:38.040023 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" event={"ID":"51b09322-4337-4ff7-b0e4-bb3cc82436d5","Type":"ContainerDied","Data":"8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b"} Nov 03 00:58:38.040434 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:38.040351 5144 scope.go:122] "RemoveContainer" containerID="8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b" Nov 03 00:58:38.042259 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:38.042231 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:38.042331 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:38.042315 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:38.042383 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:38.042368 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:58:39.051706 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:39.051674 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" event={"ID":"51b09322-4337-4ff7-b0e4-bb3cc82436d5","Type":"ContainerStarted","Data":"48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c"} Nov 03 00:58:39.958217 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:39.958186 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:58:39.958513 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:39.958485 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:58:39.958589 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:39.958565 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 00:58:40.697245 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.697210 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g"] Nov 03 00:58:40.703026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.703003 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.754165 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.754136 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g"] Nov 03 00:58:40.755265 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.755238 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-642e6a\"" Nov 03 00:58:40.787588 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.787557 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-642e6a-dockercfg-qv48l\"" Nov 03 00:58:40.788564 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.788545 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f8vg\" (UniqueName: \"kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg\") pod \"payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.788708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.788691 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.890372 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.890332 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f8vg\" (UniqueName: \"kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg\") pod \"payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.890541 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.890445 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.891222 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.891196 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.946982 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.946949 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f8vg\" (UniqueName: \"kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg\") pod \"payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:40.959059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.958999 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:58:40.959181 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:40.959053 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 00:58:41.060144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:41.060116 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:41.469867 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:41.469838 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g"] Nov 03 00:58:41.472855 ip-10-0-143-162 kubenswrapper[5144]: W1103 00:58:41.472829 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44fe2b24_5688_4e2e_b910_b0f03c173953.slice/crio-9ab8ca763656597fab899e0c656ffe7ac25f51d9c836d21e40520c3f78724d66 WatchSource:0}: Error finding container 9ab8ca763656597fab899e0c656ffe7ac25f51d9c836d21e40520c3f78724d66: Status 404 returned error can't find the container with id 9ab8ca763656597fab899e0c656ffe7ac25f51d9c836d21e40520c3f78724d66 Nov 03 00:58:41.959059 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:41.959024 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:58:41.959527 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:41.959086 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 00:58:42.082191 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:42.082156 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" event={"ID":"44fe2b24-5688-4e2e-b910-b0f03c173953","Type":"ContainerStarted","Data":"855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a"} Nov 03 00:58:42.082390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:42.082211 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" event={"ID":"44fe2b24-5688-4e2e-b910-b0f03c173953","Type":"ContainerStarted","Data":"9ab8ca763656597fab899e0c656ffe7ac25f51d9c836d21e40520c3f78724d66"} Nov 03 00:58:42.082390 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:42.082238 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:58:42.170994 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:58:42.170958 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:58:42.199709 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:42.199663 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" podStartSLOduration=2.199646607 podStartE2EDuration="2.199646607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:42.195146716 +0000 UTC m=+3212.340841171" watchObservedRunningTime="2026-11-03 00:58:42.199646607 +0000 UTC m=+3212.345341061" Nov 03 00:58:42.958921 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:42.958885 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body= Nov 03 00:58:42.959117 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:42.958934 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" Nov 03 00:58:44.049908 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:44.049873 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:58:44.050514 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:44.050491 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:58:44.051624 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:44.051603 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 00:58:46.805349 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:46.805316 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" podUID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" containerName="main" containerID="cri-o://9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805" gracePeriod=45 Nov 03 00:58:46.891502 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:46.891470 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" podUID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.254:8000/ready\": dial tcp 10.132.0.254:8000: connect: connection refused" Nov 03 00:58:47.069708 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.069652 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:58:47.123416 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.123386 5144 generic.go:356] "Generic (PLEG): container finished" podID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" containerID="9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805" exitCode=0 Nov 03 00:58:47.123552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.123433 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" Nov 03 00:58:47.123552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.123474 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" event={"ID":"891cc89c-7f89-4b9f-bea2-bb515e50e67d","Type":"ContainerDied","Data":"9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805"} Nov 03 00:58:47.123552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.123517 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf" event={"ID":"891cc89c-7f89-4b9f-bea2-bb515e50e67d","Type":"ContainerDied","Data":"5b939a256df52a9714240799997cc2b16007bad62c397e9bd85163916f23de9c"} Nov 03 00:58:47.123552 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.123537 5144 scope.go:122] "RemoveContainer" containerID="9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805" Nov 03 00:58:47.134294 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.134279 5144 scope.go:122] "RemoveContainer" containerID="9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805" Nov 03 00:58:47.134533 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:58:47.134515 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805\": container with ID starting with 9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805 not found: ID does not exist" containerID="9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805" Nov 03 00:58:47.134626 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.134539 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805"} err="failed to get container status \"9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805\": rpc error: code = NotFound desc = could not find container \"9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805\": container with ID starting with 9b13205a053d766581b1a240fe0ca74ea57cb5481fdf43ad9814cff722db1805 not found: ID does not exist" Nov 03 00:58:47.182142 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182118 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir\") pod \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " Nov 03 00:58:47.182421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182229 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs\" (UniqueName: \"kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs\") pod \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " Nov 03 00:58:47.182421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182289 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home\") pod \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " Nov 03 00:58:47.182421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182349 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm\") pod \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " Nov 03 00:58:47.182421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182364 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir" pod "891cc89c-7f89-4b9f-bea2-bb515e50e67d" (UID: "891cc89c-7f89-4b9f-bea2-bb515e50e67d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:47.182421 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182408 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache\") pod \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " Nov 03 00:58:47.182922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182456 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76\" (UniqueName: \"kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76\") pod \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\" (UID: \"891cc89c-7f89-4b9f-bea2-bb515e50e67d\") " Nov 03 00:58:47.182922 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182561 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home" pod "891cc89c-7f89-4b9f-bea2-bb515e50e67d" (UID: "891cc89c-7f89-4b9f-bea2-bb515e50e67d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:47.183026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182940 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.183026 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.182963 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.183317 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.183257 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache" pod "891cc89c-7f89-4b9f-bea2-bb515e50e67d" (UID: "891cc89c-7f89-4b9f-bea2-bb515e50e67d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:47.186959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.186936 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs" pod "891cc89c-7f89-4b9f-bea2-bb515e50e67d" (UID: "891cc89c-7f89-4b9f-bea2-bb515e50e67d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 00:58:47.186959 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.186951 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76" pod "891cc89c-7f89-4b9f-bea2-bb515e50e67d" (UID: "891cc89c-7f89-4b9f-bea2-bb515e50e67d"). InnerVolumeSpecName "kube-api-access-q8t76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:58:47.187173 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.187151 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm" pod "891cc89c-7f89-4b9f-bea2-bb515e50e67d" (UID: "891cc89c-7f89-4b9f-bea2-bb515e50e67d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 00:58:47.284561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.284538 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8t76\" (UniqueName: \"kubernetes.io/projected/891cc89c-7f89-4b9f-bea2-bb515e50e67d-kube-api-access-q8t76\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.284561 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.284560 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/891cc89c-7f89-4b9f-bea2-bb515e50e67d-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.284742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.284570 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.284742 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.284578 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/891cc89c-7f89-4b9f-bea2-bb515e50e67d-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:58:47.545039 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.545000 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf"] Nov 03 00:58:47.548943 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.548916 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-c87e2f/auth-test-model-c87e2f-kserve-77bd4df86f-bjvpf"] Nov 03 00:58:47.996990 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:47.996958 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" path="/var/lib/kubelet/pods/891cc89c-7f89-4b9f-bea2-bb515e50e67d/volumes" Nov 03 00:58:53.094207 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:58:53.094178 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 00:59:12.156341 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:59:12.156305 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:59:42.162706 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:59:42.162663 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 00:59:44.642844 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:44.642808 5144 generic.go:356] "Generic (PLEG): container finished" podID="ddceed9b-f049-40aa-9db7-246a770583d4" containerID="e8239d2dafdfaf06e919bcad0188df012baac4ef850163254ded98533a676106" exitCode=0 Nov 03 00:59:44.643371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:44.642885 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" event={"ID":"ddceed9b-f049-40aa-9db7-246a770583d4","Type":"ContainerDied","Data":"e8239d2dafdfaf06e919bcad0188df012baac4ef850163254ded98533a676106"} Nov 03 00:59:44.643371 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:44.643339 5144 scope.go:122] "RemoveContainer" containerID="e8239d2dafdfaf06e919bcad0188df012baac4ef850163254ded98533a676106" Nov 03 00:59:44.645426 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:44.645393 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:59:44.645524 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:44.645483 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:59:44.645575 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:44.645529 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 00:59:45.659205 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:45.659170 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" event={"ID":"ddceed9b-f049-40aa-9db7-246a770583d4","Type":"ContainerStarted","Data":"0920dfa2603b50b5edf4c4193ecb464d3fde40239c087d01916062bf4a885adc"} Nov 03 00:59:46.273194 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:46.273156 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:59:46.274693 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:46.274666 5144 patch_prober.go:29] interesting pod/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 00:59:46.274831 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:46.274723 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 00:59:47.273276 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:47.273241 5144 patch_prober.go:29] interesting pod/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 00:59:47.273717 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:47.273304 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 00:59:48.273306 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:48.273271 5144 patch_prober.go:29] interesting pod/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 00:59:48.273761 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:48.273337 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 00:59:48.598024 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:48.597940 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:59:49.272499 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:49.272460 5144 patch_prober.go:29] interesting pod/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 00:59:49.272705 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:49.272513 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 00:59:50.272555 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:50.272520 5144 patch_prober.go:29] interesting pod/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" start-of-body= Nov 03 00:59:50.274621 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:50.274181 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.14:15021/healthz/ready\": dial tcp 10.132.1.14:15021: connect: connection refused" Nov 03 00:59:51.277521 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:51.277484 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:59:51.278745 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:51.278723 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 00:59:52.374042 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:52.374012 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8"] Nov 03 00:59:52.484408 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:52.484376 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb"] Nov 03 00:59:57.374567 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.374526 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" podUID="38d9c691-6f27-40d2-be43-3b1ecd24526c" containerName="payload-processing" containerID="cri-o://f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453" gracePeriod=25 Nov 03 00:59:57.485154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.485124 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" podUID="9e78b96a-9903-4fd9-80d1-752cdd022a17" containerName="payload-pre-processing" containerID="cri-o://0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede" gracePeriod=25 Nov 03 00:59:57.653324 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.653296 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:59:57.689994 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.689970 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume\") pod \"38d9c691-6f27-40d2-be43-3b1ecd24526c\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " Nov 03 00:59:57.690157 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.690070 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r\" (UniqueName: \"kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r\") pod \"38d9c691-6f27-40d2-be43-3b1ecd24526c\" (UID: \"38d9c691-6f27-40d2-be43-3b1ecd24526c\") " Nov 03 00:59:57.690392 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.690362 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume" pod "38d9c691-6f27-40d2-be43-3b1ecd24526c" (UID: "38d9c691-6f27-40d2-be43-3b1ecd24526c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:57.692613 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.692585 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r" pod "38d9c691-6f27-40d2-be43-3b1ecd24526c" (UID: "38d9c691-6f27-40d2-be43-3b1ecd24526c"). InnerVolumeSpecName "kube-api-access-66d5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:57.753172 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.753141 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:59:57.761261 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.761232 5144 generic.go:356] "Generic (PLEG): container finished" podID="38d9c691-6f27-40d2-be43-3b1ecd24526c" containerID="f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453" exitCode=0 Nov 03 00:59:57.761407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.761275 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" Nov 03 00:59:57.761407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.761321 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" event={"ID":"38d9c691-6f27-40d2-be43-3b1ecd24526c","Type":"ContainerDied","Data":"f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453"} Nov 03 00:59:57.761407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.761371 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8" event={"ID":"38d9c691-6f27-40d2-be43-3b1ecd24526c","Type":"ContainerDied","Data":"97eeedbddfec74ae49776f296126d7106c40d5d82ff85367ca8bbed3ad6c1e9a"} Nov 03 00:59:57.761407 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.761393 5144 scope.go:122] "RemoveContainer" containerID="f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453" Nov 03 00:59:57.762953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.762875 5144 generic.go:356] "Generic (PLEG): container finished" podID="9e78b96a-9903-4fd9-80d1-752cdd022a17" containerID="0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede" exitCode=0 Nov 03 00:59:57.762953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.762924 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" event={"ID":"9e78b96a-9903-4fd9-80d1-752cdd022a17","Type":"ContainerDied","Data":"0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede"} Nov 03 00:59:57.762953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.762959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" event={"ID":"9e78b96a-9903-4fd9-80d1-752cdd022a17","Type":"ContainerDied","Data":"9d4c5965d848d66e0536714447e4da1a29d64a27a255f1144e4bc423642c7995"} Nov 03 00:59:57.762953 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.763018 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb" Nov 03 00:59:57.776774 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.776751 5144 scope.go:122] "RemoveContainer" containerID="f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453" Nov 03 00:59:57.777071 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:59:57.777044 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453\": container with ID starting with f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453 not found: ID does not exist" containerID="f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453" Nov 03 00:59:57.777154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.777075 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453"} err="failed to get container status \"f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453\": rpc error: code = NotFound desc = could not find container \"f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453\": container with ID starting with f12058d34bc9104393abc0e2db376137eced1c53dfb5d9d09e14b603b0a66453 not found: ID does not exist" Nov 03 00:59:57.777154 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.777094 5144 scope.go:122] "RemoveContainer" containerID="0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede" Nov 03 00:59:57.790967 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.790941 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume\") pod \"9e78b96a-9903-4fd9-80d1-752cdd022a17\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " Nov 03 00:59:57.791101 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.791018 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf\" (UniqueName: \"kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf\") pod \"9e78b96a-9903-4fd9-80d1-752cdd022a17\" (UID: \"9e78b96a-9903-4fd9-80d1-752cdd022a17\") " Nov 03 00:59:57.791387 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.791353 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume" pod "9e78b96a-9903-4fd9-80d1-752cdd022a17" (UID: "9e78b96a-9903-4fd9-80d1-752cdd022a17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 00:59:57.791592 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.791570 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38d9c691-6f27-40d2-be43-3b1ecd24526c-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:59:57.791722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.791595 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-66d5r\" (UniqueName: \"kubernetes.io/projected/38d9c691-6f27-40d2-be43-3b1ecd24526c-kube-api-access-66d5r\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:59:57.791722 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.791609 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e78b96a-9903-4fd9-80d1-752cdd022a17-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:59:57.793809 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.793784 5144 scope.go:122] "RemoveContainer" containerID="0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede" Nov 03 00:59:57.793915 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.793832 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf" pod "9e78b96a-9903-4fd9-80d1-752cdd022a17" (UID: "9e78b96a-9903-4fd9-80d1-752cdd022a17"). InnerVolumeSpecName "kube-api-access-4d4bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 00:59:57.794113 ip-10-0-143-162 kubenswrapper[5144]: E1103 00:59:57.794093 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede\": container with ID starting with 0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede not found: ID does not exist" containerID="0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede" Nov 03 00:59:57.794177 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.794125 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede"} err="failed to get container status \"0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede\": rpc error: code = NotFound desc = could not find container \"0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede\": container with ID starting with 0ad8cca5bbb305d25cf8ebce55a221a7ae9e1df36c9dbf050bb364762ce57ede not found: ID does not exist" Nov 03 00:59:57.893144 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:57.893076 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4d4bf\" (UniqueName: \"kubernetes.io/projected/9e78b96a-9903-4fd9-80d1-752cdd022a17-kube-api-access-4d4bf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 00:59:58.041467 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:58.041442 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8"] Nov 03 00:59:58.046553 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:58.046528 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c19567-78755c9db5-ltbb8"] Nov 03 00:59:58.178475 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:58.178404 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb"] Nov 03 00:59:58.182086 ip-10-0-143-162 kubenswrapper[5144]: I1103 00:59:58.182065 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-c19567-79564cfbc-w2vtb"] Nov 03 01:00:00.009200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.009169 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d9c691-6f27-40d2-be43-3b1ecd24526c" path="/var/lib/kubelet/pods/38d9c691-6f27-40d2-be43-3b1ecd24526c/volumes" Nov 03 01:00:00.009731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.009604 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e78b96a-9903-4fd9-80d1-752cdd022a17" path="/var/lib/kubelet/pods/9e78b96a-9903-4fd9-80d1-752cdd022a17/volumes" Nov 03 01:00:00.165801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.165771 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv"] Nov 03 01:00:00.166432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166418 5144 cpu_manager.go:414] "containerMap: removing container" podUID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" containerName="main" Nov 03 01:00:00.166485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166433 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" containerName="main" Nov 03 01:00:00.166485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166459 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9e78b96a-9903-4fd9-80d1-752cdd022a17" containerName="payload-pre-processing" Nov 03 01:00:00.166485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166464 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e78b96a-9903-4fd9-80d1-752cdd022a17" containerName="payload-pre-processing" Nov 03 01:00:00.166485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166483 5144 cpu_manager.go:414] "containerMap: removing container" podUID="38d9c691-6f27-40d2-be43-3b1ecd24526c" containerName="payload-processing" Nov 03 01:00:00.166617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166487 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d9c691-6f27-40d2-be43-3b1ecd24526c" containerName="payload-processing" Nov 03 01:00:00.166617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166581 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d9c691-6f27-40d2-be43-3b1ecd24526c" containerName="payload-processing" Nov 03 01:00:00.166617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166596 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="891cc89c-7f89-4b9f-bea2-bb515e50e67d" containerName="main" Nov 03 01:00:00.166617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.166605 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e78b96a-9903-4fd9-80d1-752cdd022a17" containerName="payload-pre-processing" Nov 03 01:00:00.170292 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.170272 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.173953 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.173937 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds"] Nov 03 01:00:00.177436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.177415 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.178104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.177947 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74"] Nov 03 01:00:00.180970 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.180950 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.215302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.215277 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4"] Nov 03 01:00:00.219119 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.219092 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.219248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.219140 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtnw\" (UniqueName: \"kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.219642 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.219616 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.220517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.220498 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc"] Nov 03 01:00:00.223647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.223615 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.225175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.225150 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf"] Nov 03 01:00:00.228292 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.228274 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.230277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.230257 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s"] Nov 03 01:00:00.234244 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.234225 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.235969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.235950 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7"] Nov 03 01:00:00.239888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.239857 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.241493 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.241471 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f"] Nov 03 01:00:00.245836 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.245818 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.261321 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.261297 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:00:00.265259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.265242 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg"] Nov 03 01:00:00.268340 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.268318 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.269104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.269084 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx"] Nov 03 01:00:00.272183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.272164 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.283095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.283081 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74"] Nov 03 01:00:00.288549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.288533 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv"] Nov 03 01:00:00.292733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.292715 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4"] Nov 03 01:00:00.296530 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.296512 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c"] Nov 03 01:00:00.300066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.300019 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.300495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.300476 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2"] Nov 03 01:00:00.308046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.308025 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.316046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.316023 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h"] Nov 03 01:00:00.320485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320417 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-bvg9f\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320491 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfcvr\" (UniqueName: \"kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320542 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zftn\" (UniqueName: \"kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320585 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320644 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320683 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320721 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320748 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.320770 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtnw\" (UniqueName: \"kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.323160 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.323133 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.331151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.331123 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8sqb\" (UniqueName: \"kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.331408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.331385 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.331591 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.331571 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.331777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.331757 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.331901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.331887 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzjgp\" (UniqueName: \"kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp\") pod \"maas-api-key-cleanup-29894460-bvg9f\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.332096 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332059 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh24v\" (UniqueName: \"kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.332327 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332301 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.332430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332348 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qszqb\" (UniqueName: \"kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.332430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332397 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxzt\" (UniqueName: \"kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.332430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332421 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jdvp\" (UniqueName: \"kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.332623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332499 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdcj9\" (UniqueName: \"kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.332623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332611 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.332756 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332679 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.332991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.332975 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.334804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.334783 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f"] Nov 03 01:00:00.381884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.371171 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds"] Nov 03 01:00:00.381884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.377183 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s"] Nov 03 01:00:00.383693 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.383670 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7"] Nov 03 01:00:00.392752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.392725 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtnw\" (UniqueName: \"kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.395264 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.395242 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c"] Nov 03 01:00:00.401873 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.401847 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2"] Nov 03 01:00:00.407736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.407712 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf"] Nov 03 01:00:00.412237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.412217 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc"] Nov 03 01:00:00.417863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.417844 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx"] Nov 03 01:00:00.434139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434116 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.434251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434169 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.434251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434214 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.434251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434245 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzjgp\" (UniqueName: \"kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp\") pod \"maas-api-key-cleanup-29894460-bvg9f\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.434443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434278 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh24v\" (UniqueName: \"kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.434538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434326 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.434694 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434674 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbzdx\" (UniqueName: \"kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.434836 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434820 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qszqb\" (UniqueName: \"kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.434964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434947 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxzt\" (UniqueName: \"kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.435084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435066 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jdvp\" (UniqueName: \"kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.435215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435199 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdcj9\" (UniqueName: \"kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.435359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.434975 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.435444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435013 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.435444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435337 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.435651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435607 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.435801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435776 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.435909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435822 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxm2\" (UniqueName: \"kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.435909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.435865 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.436094 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436074 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-bvg9f\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.436242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436223 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfcvr\" (UniqueName: \"kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.436376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436359 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zftn\" (UniqueName: \"kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.436581 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436561 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.436783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436721 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.436877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436732 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7wfs\" (UniqueName: \"kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.436877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436578 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.436877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436766 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894460-bvg9f\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.436877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436831 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.437095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436876 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.437095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436923 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.437095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436954 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.437095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.436998 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.437095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.437034 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8sqb\" (UniqueName: \"kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.437358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.437329 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.437742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.437716 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.437884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.437863 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.437995 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.437975 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.449787 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.449761 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg"] Nov 03 01:00:00.464647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.464462 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h"] Nov 03 01:00:00.487160 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.484088 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:00.494671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.494646 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfcvr\" (UniqueName: \"kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.496088 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.496062 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.497813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.497785 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh24v\" (UniqueName: \"kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v\") pod \"maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.499728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.498487 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxzt\" (UniqueName: \"kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.499728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.498704 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jdvp\" (UniqueName: \"kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.499728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.498923 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdcj9\" (UniqueName: \"kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.499728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.499036 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:00.500019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.499926 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzjgp\" (UniqueName: \"kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp\") pod \"maas-api-key-cleanup-29894460-bvg9f\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.500140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.500114 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8sqb\" (UniqueName: \"kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.500229 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.500124 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zftn\" (UniqueName: \"kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.507433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.507046 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qszqb\" (UniqueName: \"kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb\") pod \"maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.538428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.538394 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7wfs\" (UniqueName: \"kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.538503 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.538435 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.538503 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.538486 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.538747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.538620 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbzdx\" (UniqueName: \"kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.538747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.538690 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.538747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.538731 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxm2\" (UniqueName: \"kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.540093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.539757 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.540093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.539893 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.540093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.539908 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:00.540093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.539893 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.548509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.548342 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:00.559653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.559493 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:00.568926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.568681 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:00.577593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.577407 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:00.589149 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.588943 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:00.598728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.598370 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:00.609541 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.609106 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:00.633619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.633527 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxm2\" (UniqueName: \"kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.638378 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.638266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbzdx\" (UniqueName: \"kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.645434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.644958 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7wfs\" (UniqueName: \"kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs\") pod \"maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.645434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.645138 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:00.671052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.670841 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:00.681947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.681873 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:00.796655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.793951 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:00.968857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:00.968830 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv"] Nov 03 01:00:01.302124 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:00:01.302092 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9ec3af2_15dc_4615_b002_e466c99665ea.slice/crio-d2366111a9b1178f51e0f24b901478a8c46305fbb3ba909d011e95a214f32416 WatchSource:0}: Error finding container d2366111a9b1178f51e0f24b901478a8c46305fbb3ba909d011e95a214f32416: Status 404 returned error can't find the container with id d2366111a9b1178f51e0f24b901478a8c46305fbb3ba909d011e95a214f32416 Nov 03 01:00:01.303442 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.303422 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7"] Nov 03 01:00:01.334173 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.334148 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74"] Nov 03 01:00:01.341081 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:00:01.341051 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6241e04a_421b_43e0_aac3_533553ec9b35.slice/crio-825f259607bf84a82a2bcf7d34b4e53f9761287d0676a1c82a2ef231f7c3b6bc WatchSource:0}: Error finding container 825f259607bf84a82a2bcf7d34b4e53f9761287d0676a1c82a2ef231f7c3b6bc: Status 404 returned error can't find the container with id 825f259607bf84a82a2bcf7d34b4e53f9761287d0676a1c82a2ef231f7c3b6bc Nov 03 01:00:01.366248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.366224 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc"] Nov 03 01:00:01.379331 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.379308 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4"] Nov 03 01:00:01.403980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.403888 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg"] Nov 03 01:00:01.440666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.440615 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf"] Nov 03 01:00:01.464438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.464412 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c"] Nov 03 01:00:01.483121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.483099 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s"] Nov 03 01:00:01.501003 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.500982 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f"] Nov 03 01:00:01.505179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.505157 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2"] Nov 03 01:00:01.518603 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:00:01.518171 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5285dd_5ee8_4b87_bbc2_fd9b4870343f.slice/crio-372b0baac18e9306d587f9a20f3ad178da2c6bd6dd346e2affa0a900f5826cb6 WatchSource:0}: Error finding container 372b0baac18e9306d587f9a20f3ad178da2c6bd6dd346e2affa0a900f5826cb6: Status 404 returned error can't find the container with id 372b0baac18e9306d587f9a20f3ad178da2c6bd6dd346e2affa0a900f5826cb6 Nov 03 01:00:01.522243 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:00:01.522222 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda022f997_f8e8_4ec5_9438_9b69b291591b.slice/crio-f70f89bdaf35f01880e198a1ec150d50d7c4f73a410d5dc2e28a44ad200035a7 WatchSource:0}: Error finding container f70f89bdaf35f01880e198a1ec150d50d7c4f73a410d5dc2e28a44ad200035a7: Status 404 returned error can't find the container with id f70f89bdaf35f01880e198a1ec150d50d7c4f73a410d5dc2e28a44ad200035a7 Nov 03 01:00:01.526108 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:00:01.526082 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62ba0db4_a7a4_455d_a64a_a12879b89dd0.slice/crio-fa91b918abbe8008d7afc7554a0fe209fea2c9c77a70e6743dcc31c30559bcd8 WatchSource:0}: Error finding container fa91b918abbe8008d7afc7554a0fe209fea2c9c77a70e6743dcc31c30559bcd8: Status 404 returned error can't find the container with id fa91b918abbe8008d7afc7554a0fe209fea2c9c77a70e6743dcc31c30559bcd8 Nov 03 01:00:01.577936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.577907 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h"] Nov 03 01:00:01.586659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.586617 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds"] Nov 03 01:00:01.685941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.685916 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx"] Nov 03 01:00:01.824610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.824575 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerStarted","Data":"17d5f8ff67bbb50e6db117e33a152d9f9294de6f938d00984165ec3ea0e5e7ce"} Nov 03 01:00:01.828662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.828613 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerStarted","Data":"f70f89bdaf35f01880e198a1ec150d50d7c4f73a410d5dc2e28a44ad200035a7"} Nov 03 01:00:01.830188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.830157 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerStarted","Data":"fa91b918abbe8008d7afc7554a0fe209fea2c9c77a70e6743dcc31c30559bcd8"} Nov 03 01:00:01.831834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.831809 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" event={"ID":"b1f3731f-2cb5-4e76-8fca-36344dfa207e","Type":"ContainerStarted","Data":"0b3d919df710bbd8f8179c30fd15c27f5501f863ff97170057ec0cd5b5eee6c3"} Nov 03 01:00:01.839032 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.839002 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerStarted","Data":"6a24788c951cf44859c91e697579a284fa0cdf337b35ffc3cb21640f9f2e9a1f"} Nov 03 01:00:01.840763 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.840737 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerStarted","Data":"df78ea3b93d51fe140f2cd6637ff386210b86fabc60fc702364ec8846ec35e57"} Nov 03 01:00:01.842886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.842860 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerStarted","Data":"825f259607bf84a82a2bcf7d34b4e53f9761287d0676a1c82a2ef231f7c3b6bc"} Nov 03 01:00:01.844867 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.844754 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" event={"ID":"f9ec3af2-15dc-4615-b002-e466c99665ea","Type":"ContainerStarted","Data":"d2366111a9b1178f51e0f24b901478a8c46305fbb3ba909d011e95a214f32416"} Nov 03 01:00:01.846478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.846441 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerStarted","Data":"05099b2e0ce8a796dd541b3247e3a4b970aae19fe5035f1f1b0d999e4426aa5c"} Nov 03 01:00:01.848532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.848504 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerStarted","Data":"4e5c40afeec292937a469bc4e9932d42c94c437020337f3316a52bc94959469c"} Nov 03 01:00:01.848684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.848551 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerStarted","Data":"e624aec990b46fccba6f6ccb5d050c03225da15e725e9ca5065f39a6a99cca87"} Nov 03 01:00:01.850124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.850059 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerStarted","Data":"9d0b36db7cbc9d9929929141ab361fdb2b46a21e8212dcd2e8fdea1e33d0f802"} Nov 03 01:00:01.856710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.856686 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerStarted","Data":"bf3ddaf537f8b50c7c1208c76fa587984f8ce46d4f1feecb4bbde13981830e0f"} Nov 03 01:00:01.857942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.857922 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerStarted","Data":"372b0baac18e9306d587f9a20f3ad178da2c6bd6dd346e2affa0a900f5826cb6"} Nov 03 01:00:01.916315 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:00:01.916280 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb8028da_54bb_4a91_9e23_78ba482eafcd.slice/crio-7be957bd91058385b96d57cb20e76de063ee5d9e5dff4663f0554254567e299b WatchSource:0}: Error finding container 7be957bd91058385b96d57cb20e76de063ee5d9e5dff4663f0554254567e299b: Status 404 returned error can't find the container with id 7be957bd91058385b96d57cb20e76de063ee5d9e5dff4663f0554254567e299b Nov 03 01:00:01.966167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:01.966127 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" podStartSLOduration=1.966112756 podStartE2EDuration="1.966112756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:01.965390304 +0000 UTC m=+3292.111084761" watchObservedRunningTime="2026-11-03 01:00:01.966112756 +0000 UTC m=+3292.111807212" Nov 03 01:00:02.869278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.869243 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerStarted","Data":"5eff9c561967216845ca389256afc624013b47238d7665c5d37253b017fd26fe"} Nov 03 01:00:02.871176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.871148 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerStarted","Data":"1b104d3a97f8fb57b69ea460eb57d378205d15f43651bdde6a79893cc987f389"} Nov 03 01:00:02.872990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.872959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerStarted","Data":"63e02b40be25d27a4578eda933f320f2bf59a0958a8252ea5064a1b398185fba"} Nov 03 01:00:02.874571 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.874547 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerStarted","Data":"245171211a4dfd83c34370e648e0c8bb6e2da2b9c17d6985254808ee0dd5477c"} Nov 03 01:00:02.876364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.876336 5144 generic.go:356] "Generic (PLEG): container finished" podID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerID="50e8f387cd741b56ec0e01969d06b0d129e3087abc7886abb7aff71a832e71f7" exitCode=60 Nov 03 01:00:02.876477 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.876405 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" event={"ID":"b1f3731f-2cb5-4e76-8fca-36344dfa207e","Type":"ContainerDied","Data":"50e8f387cd741b56ec0e01969d06b0d129e3087abc7886abb7aff71a832e71f7"} Nov 03 01:00:02.876675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.876618 5144 scope.go:122] "RemoveContainer" containerID="50e8f387cd741b56ec0e01969d06b0d129e3087abc7886abb7aff71a832e71f7" Nov 03 01:00:02.878115 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.878091 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerStarted","Data":"b11218ce8ac2c594a01da1a1f2000308edcef2f812072db619e4b9284c991376"} Nov 03 01:00:02.878211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.878128 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerStarted","Data":"7be957bd91058385b96d57cb20e76de063ee5d9e5dff4663f0554254567e299b"} Nov 03 01:00:02.880265 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.880241 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerStarted","Data":"3097bd09bd94bc84113ae4742baffac3beb11c7d10fbfd3cbb1e168a03baeccc"} Nov 03 01:00:02.881996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.881911 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerStarted","Data":"69b764c62f13067c2a2298fcca00d9934456655f1c565d26bb4e7933a6b43ea7"} Nov 03 01:00:02.883685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.883660 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerStarted","Data":"b57b5c56a23ecb04ad822313fed03db4888937495190f86281685722b6100706"} Nov 03 01:00:02.885304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.885283 5144 generic.go:356] "Generic (PLEG): container finished" podID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerID="e0d0eec78c03e48d6fc8378928b6d2c1c7b974a0a2922d8a298154ee82fe7c59" exitCode=60 Nov 03 01:00:02.885383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.885360 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" event={"ID":"f9ec3af2-15dc-4615-b002-e466c99665ea","Type":"ContainerDied","Data":"e0d0eec78c03e48d6fc8378928b6d2c1c7b974a0a2922d8a298154ee82fe7c59"} Nov 03 01:00:02.885524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.885508 5144 scope.go:122] "RemoveContainer" containerID="e0d0eec78c03e48d6fc8378928b6d2c1c7b974a0a2922d8a298154ee82fe7c59" Nov 03 01:00:02.886826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.886807 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerStarted","Data":"0757df626ab61448a80311c2792797a67a8320bedf245739c913822428b67291"} Nov 03 01:00:02.888271 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.888251 5144 generic.go:356] "Generic (PLEG): container finished" podID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerID="4e5c40afeec292937a469bc4e9932d42c94c437020337f3316a52bc94959469c" exitCode=60 Nov 03 01:00:02.888351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.888335 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerDied","Data":"4e5c40afeec292937a469bc4e9932d42c94c437020337f3316a52bc94959469c"} Nov 03 01:00:02.888524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.888511 5144 scope.go:122] "RemoveContainer" containerID="4e5c40afeec292937a469bc4e9932d42c94c437020337f3316a52bc94959469c" Nov 03 01:00:02.889980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.889954 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerStarted","Data":"f22754c6bd84349685f5e592fc0c72a3088005392b133435b44919c7f0a4826a"} Nov 03 01:00:02.891469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.891448 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerStarted","Data":"7e48f43dac83e7c745a1240179493cc89c32ec1bc084518ffb914d0830a51c9f"} Nov 03 01:00:02.985595 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:02.985519 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" podStartSLOduration=2.985505115 podStartE2EDuration="2.985505115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.98373416 +0000 UTC m=+3293.129428612" watchObservedRunningTime="2026-11-03 01:00:02.985505115 +0000 UTC m=+3293.131199566" Nov 03 01:00:03.098769 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.098725 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" podStartSLOduration=3.098711587 podStartE2EDuration="3.098711587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.095303038 +0000 UTC m=+3293.240997495" watchObservedRunningTime="2026-11-03 01:00:03.098711587 +0000 UTC m=+3293.244406044" Nov 03 01:00:03.207127 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.207073 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" podStartSLOduration=3.207055517 podStartE2EDuration="3.207055517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.206982165 +0000 UTC m=+3293.352676634" watchObservedRunningTime="2026-11-03 01:00:03.207055517 +0000 UTC m=+3293.352749996" Nov 03 01:00:03.321708 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.321662 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" podStartSLOduration=3.321623548 podStartE2EDuration="3.321623548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.319820731 +0000 UTC m=+3293.465515188" watchObservedRunningTime="2026-11-03 01:00:03.321623548 +0000 UTC m=+3293.467318053" Nov 03 01:00:03.437098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.437053 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" podStartSLOduration=3.43704044 podStartE2EDuration="3.43704044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.435242948 +0000 UTC m=+3293.580937422" watchObservedRunningTime="2026-11-03 01:00:03.43704044 +0000 UTC m=+3293.582734941" Nov 03 01:00:03.554090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.554035 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" podStartSLOduration=3.5540183709999997 podStartE2EDuration="3.554018371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.546881661 +0000 UTC m=+3293.692576320" watchObservedRunningTime="2026-11-03 01:00:03.554018371 +0000 UTC m=+3293.699712827" Nov 03 01:00:03.781161 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.781069 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" podStartSLOduration=3.781055345 podStartE2EDuration="3.781055345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.77824707 +0000 UTC m=+3293.923941526" watchObservedRunningTime="2026-11-03 01:00:03.781055345 +0000 UTC m=+3293.926749797" Nov 03 01:00:03.890613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.890562 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" podStartSLOduration=3.890546604 podStartE2EDuration="3.890546604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:03.889576707 +0000 UTC m=+3294.035271187" watchObservedRunningTime="2026-11-03 01:00:03.890546604 +0000 UTC m=+3294.036241061" Nov 03 01:00:03.912426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.912386 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" event={"ID":"b1f3731f-2cb5-4e76-8fca-36344dfa207e","Type":"ContainerStarted","Data":"936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17"} Nov 03 01:00:03.914314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.914257 5144 generic.go:356] "Generic (PLEG): container finished" podID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerID="b11218ce8ac2c594a01da1a1f2000308edcef2f812072db619e4b9284c991376" exitCode=60 Nov 03 01:00:03.914688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.914381 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerDied","Data":"b11218ce8ac2c594a01da1a1f2000308edcef2f812072db619e4b9284c991376"} Nov 03 01:00:03.914775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.914715 5144 scope.go:122] "RemoveContainer" containerID="b11218ce8ac2c594a01da1a1f2000308edcef2f812072db619e4b9284c991376" Nov 03 01:00:03.918305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.918287 5144 generic.go:356] "Generic (PLEG): container finished" podID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerID="3097bd09bd94bc84113ae4742baffac3beb11c7d10fbfd3cbb1e168a03baeccc" exitCode=60 Nov 03 01:00:03.918421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.918401 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerDied","Data":"3097bd09bd94bc84113ae4742baffac3beb11c7d10fbfd3cbb1e168a03baeccc"} Nov 03 01:00:03.918602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.918582 5144 scope.go:122] "RemoveContainer" containerID="3097bd09bd94bc84113ae4742baffac3beb11c7d10fbfd3cbb1e168a03baeccc" Nov 03 01:00:03.920792 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.920770 5144 generic.go:356] "Generic (PLEG): container finished" podID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerID="69b764c62f13067c2a2298fcca00d9934456655f1c565d26bb4e7933a6b43ea7" exitCode=60 Nov 03 01:00:03.920901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.920849 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerDied","Data":"69b764c62f13067c2a2298fcca00d9934456655f1c565d26bb4e7933a6b43ea7"} Nov 03 01:00:03.921059 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.921043 5144 scope.go:122] "RemoveContainer" containerID="69b764c62f13067c2a2298fcca00d9934456655f1c565d26bb4e7933a6b43ea7" Nov 03 01:00:03.922786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.922766 5144 generic.go:356] "Generic (PLEG): container finished" podID="6241e04a-421b-43e0-aac3-533553ec9b35" containerID="b57b5c56a23ecb04ad822313fed03db4888937495190f86281685722b6100706" exitCode=60 Nov 03 01:00:03.922890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.922854 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerDied","Data":"b57b5c56a23ecb04ad822313fed03db4888937495190f86281685722b6100706"} Nov 03 01:00:03.923053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.923034 5144 scope.go:122] "RemoveContainer" containerID="b57b5c56a23ecb04ad822313fed03db4888937495190f86281685722b6100706" Nov 03 01:00:03.925057 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.925026 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" event={"ID":"f9ec3af2-15dc-4615-b002-e466c99665ea","Type":"ContainerStarted","Data":"5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8"} Nov 03 01:00:03.939414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.939388 5144 generic.go:356] "Generic (PLEG): container finished" podID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerID="0757df626ab61448a80311c2792797a67a8320bedf245739c913822428b67291" exitCode=60 Nov 03 01:00:03.939507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.939474 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerDied","Data":"0757df626ab61448a80311c2792797a67a8320bedf245739c913822428b67291"} Nov 03 01:00:03.939777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.939759 5144 scope.go:122] "RemoveContainer" containerID="0757df626ab61448a80311c2792797a67a8320bedf245739c913822428b67291" Nov 03 01:00:03.944100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.944077 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerStarted","Data":"216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89"} Nov 03 01:00:03.946480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.946456 5144 generic.go:356] "Generic (PLEG): container finished" podID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerID="f22754c6bd84349685f5e592fc0c72a3088005392b133435b44919c7f0a4826a" exitCode=60 Nov 03 01:00:03.946742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.946725 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerDied","Data":"f22754c6bd84349685f5e592fc0c72a3088005392b133435b44919c7f0a4826a"} Nov 03 01:00:03.947190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.947040 5144 scope.go:122] "RemoveContainer" containerID="f22754c6bd84349685f5e592fc0c72a3088005392b133435b44919c7f0a4826a" Nov 03 01:00:03.949035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.949016 5144 generic.go:356] "Generic (PLEG): container finished" podID="f2257916-7c4d-4956-a42e-32193b32467f" containerID="7e48f43dac83e7c745a1240179493cc89c32ec1bc084518ffb914d0830a51c9f" exitCode=60 Nov 03 01:00:03.949141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.949094 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerDied","Data":"7e48f43dac83e7c745a1240179493cc89c32ec1bc084518ffb914d0830a51c9f"} Nov 03 01:00:03.949274 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.949256 5144 scope.go:122] "RemoveContainer" containerID="7e48f43dac83e7c745a1240179493cc89c32ec1bc084518ffb914d0830a51c9f" Nov 03 01:00:03.953123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.953037 5144 generic.go:356] "Generic (PLEG): container finished" podID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerID="5eff9c561967216845ca389256afc624013b47238d7665c5d37253b017fd26fe" exitCode=60 Nov 03 01:00:03.953207 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.953127 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerDied","Data":"5eff9c561967216845ca389256afc624013b47238d7665c5d37253b017fd26fe"} Nov 03 01:00:03.953312 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.953290 5144 scope.go:122] "RemoveContainer" containerID="5eff9c561967216845ca389256afc624013b47238d7665c5d37253b017fd26fe" Nov 03 01:00:03.954925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.954905 5144 generic.go:356] "Generic (PLEG): container finished" podID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerID="1b104d3a97f8fb57b69ea460eb57d378205d15f43651bdde6a79893cc987f389" exitCode=60 Nov 03 01:00:03.955038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.954995 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerDied","Data":"1b104d3a97f8fb57b69ea460eb57d378205d15f43651bdde6a79893cc987f389"} Nov 03 01:00:03.955160 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.955144 5144 scope.go:122] "RemoveContainer" containerID="1b104d3a97f8fb57b69ea460eb57d378205d15f43651bdde6a79893cc987f389" Nov 03 01:00:03.956645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.956613 5144 generic.go:356] "Generic (PLEG): container finished" podID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerID="63e02b40be25d27a4578eda933f320f2bf59a0958a8252ea5064a1b398185fba" exitCode=60 Nov 03 01:00:03.956754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.956738 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerDied","Data":"63e02b40be25d27a4578eda933f320f2bf59a0958a8252ea5064a1b398185fba"} Nov 03 01:00:03.956993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.956975 5144 scope.go:122] "RemoveContainer" containerID="63e02b40be25d27a4578eda933f320f2bf59a0958a8252ea5064a1b398185fba" Nov 03 01:00:03.960065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.960040 5144 generic.go:356] "Generic (PLEG): container finished" podID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerID="245171211a4dfd83c34370e648e0c8bb6e2da2b9c17d6985254808ee0dd5477c" exitCode=60 Nov 03 01:00:03.960166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.960110 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerDied","Data":"245171211a4dfd83c34370e648e0c8bb6e2da2b9c17d6985254808ee0dd5477c"} Nov 03 01:00:03.960321 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:03.960306 5144 scope.go:122] "RemoveContainer" containerID="245171211a4dfd83c34370e648e0c8bb6e2da2b9c17d6985254808ee0dd5477c" Nov 03 01:00:04.009194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.009141 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" podStartSLOduration=4.009122538 podStartE2EDuration="4.009122538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.005967734 +0000 UTC m=+3294.151662211" watchObservedRunningTime="2026-11-03 01:00:04.009122538 +0000 UTC m=+3294.154816997" Nov 03 01:00:04.235473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.235422 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" podStartSLOduration=4.235408365 podStartE2EDuration="4.235408365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.234807345 +0000 UTC m=+3294.380501820" watchObservedRunningTime="2026-11-03 01:00:04.235408365 +0000 UTC m=+3294.381102821" Nov 03 01:00:04.472305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.472250 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" podStartSLOduration=4.472236204 podStartE2EDuration="4.472236204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.470386802 +0000 UTC m=+3294.616081260" watchObservedRunningTime="2026-11-03 01:00:04.472236204 +0000 UTC m=+3294.617930659" Nov 03 01:00:04.697156 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.697062 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" podStartSLOduration=4.697047199 podStartE2EDuration="4.697047199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:04.696031401 +0000 UTC m=+3294.841725875" watchObservedRunningTime="2026-11-03 01:00:04.697047199 +0000 UTC m=+3294.842741654" Nov 03 01:00:04.972208 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.972149 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerStarted","Data":"bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9"} Nov 03 01:00:04.974132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.974106 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerStarted","Data":"ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d"} Nov 03 01:00:04.976096 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.976065 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerStarted","Data":"7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5"} Nov 03 01:00:04.978085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.978065 5144 generic.go:356] "Generic (PLEG): container finished" podID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerID="5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8" exitCode=60 Nov 03 01:00:04.978177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.978131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" event={"ID":"f9ec3af2-15dc-4615-b002-e466c99665ea","Type":"ContainerDied","Data":"5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8"} Nov 03 01:00:04.978233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.978178 5144 scope.go:122] "RemoveContainer" containerID="e0d0eec78c03e48d6fc8378928b6d2c1c7b974a0a2922d8a298154ee82fe7c59" Nov 03 01:00:04.978367 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.978340 5144 scope.go:122] "RemoveContainer" containerID="5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8" Nov 03 01:00:04.978617 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:04.978592 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7_odh-ai-gateway-infra(f9ec3af2-15dc-4615-b002-e466c99665ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" Nov 03 01:00:04.980129 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.980109 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerStarted","Data":"67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d"} Nov 03 01:00:04.981962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.981936 5144 generic.go:356] "Generic (PLEG): container finished" podID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerID="216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89" exitCode=60 Nov 03 01:00:04.982069 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.982017 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerDied","Data":"216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89"} Nov 03 01:00:04.982205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.982189 5144 scope.go:122] "RemoveContainer" containerID="216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89" Nov 03 01:00:04.982449 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:04.982431 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv_odh-ai-gateway-infra(b68024ff-ae2e-4ea6-aa70-85a1816be6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" Nov 03 01:00:04.983967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.983939 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerStarted","Data":"77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524"} Nov 03 01:00:04.985896 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.985873 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerStarted","Data":"fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010"} Nov 03 01:00:04.987966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.987946 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerStarted","Data":"8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2"} Nov 03 01:00:04.990020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.990001 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerStarted","Data":"965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2"} Nov 03 01:00:04.991922 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.991904 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerStarted","Data":"b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73"} Nov 03 01:00:04.993573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.993554 5144 scope.go:122] "RemoveContainer" containerID="4e5c40afeec292937a469bc4e9932d42c94c437020337f3316a52bc94959469c" Nov 03 01:00:04.993852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.993824 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerStarted","Data":"ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01"} Nov 03 01:00:04.995866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.995845 5144 generic.go:356] "Generic (PLEG): container finished" podID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerID="936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17" exitCode=60 Nov 03 01:00:04.995940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.995907 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" event={"ID":"b1f3731f-2cb5-4e76-8fca-36344dfa207e","Type":"ContainerDied","Data":"936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17"} Nov 03 01:00:04.996070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.996056 5144 scope.go:122] "RemoveContainer" containerID="936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17" Nov 03 01:00:04.996299 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:04.996278 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4_odh-ai-gateway-infra(b1f3731f-2cb5-4e76-8fca-36344dfa207e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" Nov 03 01:00:04.997777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:04.997760 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerStarted","Data":"396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da"} Nov 03 01:00:05.009604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:05.009584 5144 scope.go:122] "RemoveContainer" containerID="50e8f387cd741b56ec0e01969d06b0d129e3087abc7886abb7aff71a832e71f7" Nov 03 01:00:05.841793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:05.841740 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" podStartSLOduration=5.841725269 podStartE2EDuration="5.841725269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:05.839454851 +0000 UTC m=+3295.985149308" watchObservedRunningTime="2026-11-03 01:00:05.841725269 +0000 UTC m=+3295.987419727" Nov 03 01:00:06.008107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.008070 5144 generic.go:356] "Generic (PLEG): container finished" podID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerID="ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01" exitCode=60 Nov 03 01:00:06.008536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.008154 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerDied","Data":"ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01"} Nov 03 01:00:06.008536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.008218 5144 scope.go:122] "RemoveContainer" containerID="245171211a4dfd83c34370e648e0c8bb6e2da2b9c17d6985254808ee0dd5477c" Nov 03 01:00:06.008536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.008351 5144 scope.go:122] "RemoveContainer" containerID="ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01" Nov 03 01:00:06.008730 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.008695 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c_odh-ai-gateway-infra(62ba0db4-a7a4-455d-a64a-a12879b89dd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" Nov 03 01:00:06.010365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.010336 5144 scope.go:122] "RemoveContainer" containerID="936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17" Nov 03 01:00:06.010621 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.010595 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4_odh-ai-gateway-infra(b1f3731f-2cb5-4e76-8fca-36344dfa207e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" Nov 03 01:00:06.012238 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.012216 5144 generic.go:356] "Generic (PLEG): container finished" podID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerID="396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da" exitCode=60 Nov 03 01:00:06.012331 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.012272 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerDied","Data":"396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da"} Nov 03 01:00:06.012490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.012474 5144 scope.go:122] "RemoveContainer" containerID="396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da" Nov 03 01:00:06.012772 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.012746 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx_odh-ai-gateway-infra(bb8028da-54bb-4a91-9e23-78ba482eafcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" Nov 03 01:00:06.014234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.014209 5144 generic.go:356] "Generic (PLEG): container finished" podID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerID="bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9" exitCode=60 Nov 03 01:00:06.014339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.014265 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerDied","Data":"bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9"} Nov 03 01:00:06.014529 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.014514 5144 scope.go:122] "RemoveContainer" containerID="bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9" Nov 03 01:00:06.014745 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.014726 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds_odh-ai-gateway-infra(c94d86b5-e3a6-4653-97bc-6456b9a77362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" Nov 03 01:00:06.016892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.016703 5144 generic.go:356] "Generic (PLEG): container finished" podID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerID="ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d" exitCode=60 Nov 03 01:00:06.016892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.016773 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerDied","Data":"ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d"} Nov 03 01:00:06.017085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.017068 5144 scope.go:122] "RemoveContainer" containerID="ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d" Nov 03 01:00:06.017326 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.017305 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s_odh-ai-gateway-infra(e9cd69ff-7d82-4774-bd9a-505db369f298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" Nov 03 01:00:06.019150 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.019131 5144 generic.go:356] "Generic (PLEG): container finished" podID="6241e04a-421b-43e0-aac3-533553ec9b35" containerID="7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5" exitCode=60 Nov 03 01:00:06.019256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.019241 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerDied","Data":"7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5"} Nov 03 01:00:06.019503 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.019486 5144 scope.go:122] "RemoveContainer" containerID="7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5" Nov 03 01:00:06.019715 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.019695 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74_odh-ai-gateway-infra(6241e04a-421b-43e0-aac3-533553ec9b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" Nov 03 01:00:06.022793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.022771 5144 scope.go:122] "RemoveContainer" containerID="5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8" Nov 03 01:00:06.023018 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.022994 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7_odh-ai-gateway-infra(f9ec3af2-15dc-4615-b002-e466c99665ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" Nov 03 01:00:06.024101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.024083 5144 scope.go:122] "RemoveContainer" containerID="b11218ce8ac2c594a01da1a1f2000308edcef2f812072db619e4b9284c991376" Nov 03 01:00:06.024751 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.024722 5144 generic.go:356] "Generic (PLEG): container finished" podID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerID="67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d" exitCode=60 Nov 03 01:00:06.024855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.024780 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerDied","Data":"67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d"} Nov 03 01:00:06.025036 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.025020 5144 scope.go:122] "RemoveContainer" containerID="67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d" Nov 03 01:00:06.025484 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.025459 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-bvg9f_odh-ai-gateway-infra(de7c42b8-71e6-461d-b8a1-5b76c12b8047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" Nov 03 01:00:06.027140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.027122 5144 scope.go:122] "RemoveContainer" containerID="216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89" Nov 03 01:00:06.027335 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.027313 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv_odh-ai-gateway-infra(b68024ff-ae2e-4ea6-aa70-85a1816be6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" Nov 03 01:00:06.028987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.028965 5144 generic.go:356] "Generic (PLEG): container finished" podID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerID="77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524" exitCode=60 Nov 03 01:00:06.029089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.029048 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerDied","Data":"77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524"} Nov 03 01:00:06.029531 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.029286 5144 scope.go:122] "RemoveContainer" containerID="77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524" Nov 03 01:00:06.029531 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.029507 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2_odh-ai-gateway-infra(d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" Nov 03 01:00:06.031277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.031204 5144 generic.go:356] "Generic (PLEG): container finished" podID="f2257916-7c4d-4956-a42e-32193b32467f" containerID="fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010" exitCode=60 Nov 03 01:00:06.031277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.031229 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerDied","Data":"fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010"} Nov 03 01:00:06.031439 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.031419 5144 scope.go:122] "RemoveContainer" containerID="fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010" Nov 03 01:00:06.031624 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.031603 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc_odh-ai-gateway-infra(f2257916-7c4d-4956-a42e-32193b32467f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" podUID="f2257916-7c4d-4956-a42e-32193b32467f" Nov 03 01:00:06.033353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.033336 5144 generic.go:356] "Generic (PLEG): container finished" podID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerID="8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2" exitCode=60 Nov 03 01:00:06.033443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.033424 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerDied","Data":"8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2"} Nov 03 01:00:06.033606 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.033589 5144 scope.go:122] "RemoveContainer" containerID="8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2" Nov 03 01:00:06.034290 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.034269 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg_odh-ai-gateway-infra(9e5285dd-5ee8-4b87-bbc2-fd9b4870343f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" Nov 03 01:00:06.035375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.035352 5144 generic.go:356] "Generic (PLEG): container finished" podID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerID="965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2" exitCode=60 Nov 03 01:00:06.035470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.035401 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerDied","Data":"965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2"} Nov 03 01:00:06.035683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.035664 5144 scope.go:122] "RemoveContainer" containerID="965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2" Nov 03 01:00:06.035896 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.035872 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h_odh-ai-gateway-infra(8c1a1243-df11-4de9-ad33-bce7df581ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" Nov 03 01:00:06.037491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.037467 5144 generic.go:356] "Generic (PLEG): container finished" podID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerID="b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73" exitCode=60 Nov 03 01:00:06.037582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.037514 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerDied","Data":"b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73"} Nov 03 01:00:06.037729 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.037715 5144 scope.go:122] "RemoveContainer" containerID="b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73" Nov 03 01:00:06.037909 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:06.037892 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf_odh-ai-gateway-infra(a022f997-f8e8-4ec5-9438-9b69b291591b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" Nov 03 01:00:06.041045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.041027 5144 scope.go:122] "RemoveContainer" containerID="3097bd09bd94bc84113ae4742baffac3beb11c7d10fbfd3cbb1e168a03baeccc" Nov 03 01:00:06.054835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.054816 5144 scope.go:122] "RemoveContainer" containerID="69b764c62f13067c2a2298fcca00d9934456655f1c565d26bb4e7933a6b43ea7" Nov 03 01:00:06.068846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.068827 5144 scope.go:122] "RemoveContainer" containerID="b57b5c56a23ecb04ad822313fed03db4888937495190f86281685722b6100706" Nov 03 01:00:06.082860 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.082841 5144 scope.go:122] "RemoveContainer" containerID="0757df626ab61448a80311c2792797a67a8320bedf245739c913822428b67291" Nov 03 01:00:06.098379 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.098359 5144 scope.go:122] "RemoveContainer" containerID="f22754c6bd84349685f5e592fc0c72a3088005392b133435b44919c7f0a4826a" Nov 03 01:00:06.111539 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.111522 5144 scope.go:122] "RemoveContainer" containerID="7e48f43dac83e7c745a1240179493cc89c32ec1bc084518ffb914d0830a51c9f" Nov 03 01:00:06.124824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.124807 5144 scope.go:122] "RemoveContainer" containerID="5eff9c561967216845ca389256afc624013b47238d7665c5d37253b017fd26fe" Nov 03 01:00:06.138272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.138247 5144 scope.go:122] "RemoveContainer" containerID="1b104d3a97f8fb57b69ea460eb57d378205d15f43651bdde6a79893cc987f389" Nov 03 01:00:06.152068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:06.152041 5144 scope.go:122] "RemoveContainer" containerID="63e02b40be25d27a4578eda933f320f2bf59a0958a8252ea5064a1b398185fba" Nov 03 01:00:07.051645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.051599 5144 scope.go:122] "RemoveContainer" containerID="ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01" Nov 03 01:00:07.052071 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.051934 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c_odh-ai-gateway-infra(62ba0db4-a7a4-455d-a64a-a12879b89dd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" Nov 03 01:00:07.053660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.053614 5144 scope.go:122] "RemoveContainer" containerID="396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da" Nov 03 01:00:07.053892 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.053861 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx_odh-ai-gateway-infra(bb8028da-54bb-4a91-9e23-78ba482eafcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" Nov 03 01:00:07.055526 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.055509 5144 scope.go:122] "RemoveContainer" containerID="bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9" Nov 03 01:00:07.055793 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.055770 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds_odh-ai-gateway-infra(c94d86b5-e3a6-4653-97bc-6456b9a77362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" Nov 03 01:00:07.057358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.057341 5144 scope.go:122] "RemoveContainer" containerID="ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d" Nov 03 01:00:07.057590 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.057566 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s_odh-ai-gateway-infra(e9cd69ff-7d82-4774-bd9a-505db369f298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" Nov 03 01:00:07.059136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.059115 5144 scope.go:122] "RemoveContainer" containerID="7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5" Nov 03 01:00:07.059386 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.059358 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74_odh-ai-gateway-infra(6241e04a-421b-43e0-aac3-533553ec9b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" Nov 03 01:00:07.062255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.062234 5144 scope.go:122] "RemoveContainer" containerID="67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d" Nov 03 01:00:07.062519 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.062498 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-bvg9f_odh-ai-gateway-infra(de7c42b8-71e6-461d-b8a1-5b76c12b8047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" Nov 03 01:00:07.063890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.063869 5144 scope.go:122] "RemoveContainer" containerID="77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524" Nov 03 01:00:07.064072 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.064051 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2_odh-ai-gateway-infra(d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" Nov 03 01:00:07.065458 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.065439 5144 scope.go:122] "RemoveContainer" containerID="fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010" Nov 03 01:00:07.065675 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.065657 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc_odh-ai-gateway-infra(f2257916-7c4d-4956-a42e-32193b32467f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" podUID="f2257916-7c4d-4956-a42e-32193b32467f" Nov 03 01:00:07.066917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.066901 5144 scope.go:122] "RemoveContainer" containerID="8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2" Nov 03 01:00:07.067108 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.067087 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg_odh-ai-gateway-infra(9e5285dd-5ee8-4b87-bbc2-fd9b4870343f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" Nov 03 01:00:07.068415 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.068397 5144 scope.go:122] "RemoveContainer" containerID="965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2" Nov 03 01:00:07.068602 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.068577 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h_odh-ai-gateway-infra(8c1a1243-df11-4de9-ad33-bce7df581ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" Nov 03 01:00:07.069879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:07.069860 5144 scope.go:122] "RemoveContainer" containerID="b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73" Nov 03 01:00:07.070065 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:07.070046 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf_odh-ai-gateway-infra(a022f997-f8e8-4ec5-9438-9b69b291591b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" Nov 03 01:00:10.774821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:10.774790 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:00:10.788569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:10.788541 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:00:11.048583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:11.048504 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h"] Nov 03 01:00:11.097771 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:11.097744 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6886ed5-dsb4h"] Nov 03 01:00:11.998739 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:11.998709 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cac8548-13a9-43ce-ba21-2c1676a5c76d" path="/var/lib/kubelet/pods/8cac8548-13a9-43ce-ba21-2c1676a5c76d/volumes" Nov 03 01:00:12.166862 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:12.166824 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:00:15.988793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:15.988758 5144 scope.go:122] "RemoveContainer" containerID="216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89" Nov 03 01:00:15.989283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:15.988879 5144 scope.go:122] "RemoveContainer" containerID="936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17" Nov 03 01:00:15.989283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:15.989033 5144 scope.go:122] "RemoveContainer" containerID="5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8" Nov 03 01:00:16.989010 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:16.988975 5144 scope.go:122] "RemoveContainer" containerID="7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5" Nov 03 01:00:16.989576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:16.989544 5144 scope.go:122] "RemoveContainer" containerID="396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da" Nov 03 01:00:16.990133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:16.989920 5144 scope.go:122] "RemoveContainer" containerID="b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73" Nov 03 01:00:16.990206 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:16.990145 5144 scope.go:122] "RemoveContainer" containerID="77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524" Nov 03 01:00:17.199807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.199766 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerStarted","Data":"474060ed7d056a0fffe116362b53a79c8bb94d0f2ea9a2c90e759d7c7e368ca6"} Nov 03 01:00:17.202543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.202518 5144 generic.go:356] "Generic (PLEG): container finished" podID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerID="8b200cf2b452fd708fab0c25e10f7331bbd1ea4bfd4002d3f21b5fb60f68006f" exitCode=60 Nov 03 01:00:17.202733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.202591 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" event={"ID":"b1f3731f-2cb5-4e76-8fca-36344dfa207e","Type":"ContainerDied","Data":"8b200cf2b452fd708fab0c25e10f7331bbd1ea4bfd4002d3f21b5fb60f68006f"} Nov 03 01:00:17.202733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.202662 5144 scope.go:122] "RemoveContainer" containerID="936d0db9a7a9a35070127181fd235180942d4658a7b851b5cb03e2ffb91c8a17" Nov 03 01:00:17.203010 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.202980 5144 scope.go:122] "RemoveContainer" containerID="8b200cf2b452fd708fab0c25e10f7331bbd1ea4bfd4002d3f21b5fb60f68006f" Nov 03 01:00:17.203306 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:17.203284 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4_odh-ai-gateway-infra(b1f3731f-2cb5-4e76-8fca-36344dfa207e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" Nov 03 01:00:17.204897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.204876 5144 generic.go:356] "Generic (PLEG): container finished" podID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerID="a1a6db201506272184f9db4a2c5a93f9a9d5ad7cfff3dc516df0b84d13120361" exitCode=60 Nov 03 01:00:17.205002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.204908 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" event={"ID":"f9ec3af2-15dc-4615-b002-e466c99665ea","Type":"ContainerDied","Data":"a1a6db201506272184f9db4a2c5a93f9a9d5ad7cfff3dc516df0b84d13120361"} Nov 03 01:00:17.205246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.205229 5144 scope.go:122] "RemoveContainer" containerID="a1a6db201506272184f9db4a2c5a93f9a9d5ad7cfff3dc516df0b84d13120361" Nov 03 01:00:17.205526 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:17.205505 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7_odh-ai-gateway-infra(f9ec3af2-15dc-4615-b002-e466c99665ea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" Nov 03 01:00:17.410920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.410893 5144 scope.go:122] "RemoveContainer" containerID="5a910eb5580a0b4690759037f82bd9f734f92379fcb910a0b56441c9a9c503a8" Nov 03 01:00:17.989574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.989483 5144 scope.go:122] "RemoveContainer" containerID="67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d" Nov 03 01:00:17.990077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.989609 5144 scope.go:122] "RemoveContainer" containerID="965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2" Nov 03 01:00:17.990077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:17.989827 5144 scope.go:122] "RemoveContainer" containerID="fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010" Nov 03 01:00:18.216956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.216925 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerStarted","Data":"078c07aae61c9e50c08d85c89159a57492ef7ae621a2d83974db1266fbe3ad53"} Nov 03 01:00:18.219414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.219390 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerStarted","Data":"ac3a89ba6265b86f5e40dd8470902fc81b5e5c22924374447725f84183320671"} Nov 03 01:00:18.223819 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.223794 5144 generic.go:356] "Generic (PLEG): container finished" podID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerID="474060ed7d056a0fffe116362b53a79c8bb94d0f2ea9a2c90e759d7c7e368ca6" exitCode=60 Nov 03 01:00:18.223956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.223905 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerDied","Data":"474060ed7d056a0fffe116362b53a79c8bb94d0f2ea9a2c90e759d7c7e368ca6"} Nov 03 01:00:18.223956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.223950 5144 scope.go:122] "RemoveContainer" containerID="216ac7183dd636c3215c64af453ec0f9605b850b7b48fe336638330f8377aa89" Nov 03 01:00:18.224252 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.224234 5144 scope.go:122] "RemoveContainer" containerID="474060ed7d056a0fffe116362b53a79c8bb94d0f2ea9a2c90e759d7c7e368ca6" Nov 03 01:00:18.224485 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:18.224464 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv_odh-ai-gateway-infra(b68024ff-ae2e-4ea6-aa70-85a1816be6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" Nov 03 01:00:18.228609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.228589 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerStarted","Data":"d6fffdab35a0e009ab9447df3c444fa4243e3d9cb5302860fc7a72155968af72"} Nov 03 01:00:18.231416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.231382 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerStarted","Data":"8c98a7bbbd77d976a1464a4ef99bde37411beb44aced9f7199f670a311d78988"} Nov 03 01:00:18.332291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.327745 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv"] Nov 03 01:00:18.436409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.436382 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4"] Nov 03 01:00:18.550461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.550433 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7"] Nov 03 01:00:18.593926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.593904 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:18.645534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.645503 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca\") pod \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " Nov 03 01:00:18.645723 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.645680 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn\" (UniqueName: \"kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn\") pod \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\" (UID: \"b1f3731f-2cb5-4e76-8fca-36344dfa207e\") " Nov 03 01:00:18.645968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.645941 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca" pod "b1f3731f-2cb5-4e76-8fca-36344dfa207e" (UID: "b1f3731f-2cb5-4e76-8fca-36344dfa207e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:18.646291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.646272 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b1f3731f-2cb5-4e76-8fca-36344dfa207e-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.648283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.648255 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn" pod "b1f3731f-2cb5-4e76-8fca-36344dfa207e" (UID: "b1f3731f-2cb5-4e76-8fca-36344dfa207e"). InnerVolumeSpecName "kube-api-access-2zftn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:18.709013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.708989 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:18.746976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.746951 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca\") pod \"f9ec3af2-15dc-4615-b002-e466c99665ea\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " Nov 03 01:00:18.747076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.747038 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb\" (UniqueName: \"kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb\") pod \"f9ec3af2-15dc-4615-b002-e466c99665ea\" (UID: \"f9ec3af2-15dc-4615-b002-e466c99665ea\") " Nov 03 01:00:18.747286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.747266 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca" pod "f9ec3af2-15dc-4615-b002-e466c99665ea" (UID: "f9ec3af2-15dc-4615-b002-e466c99665ea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:18.747583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.747558 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zftn\" (UniqueName: \"kubernetes.io/projected/b1f3731f-2cb5-4e76-8fca-36344dfa207e-kube-api-access-2zftn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.747712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.747586 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f9ec3af2-15dc-4615-b002-e466c99665ea-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.749191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.749166 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb" pod "f9ec3af2-15dc-4615-b002-e466c99665ea" (UID: "f9ec3af2-15dc-4615-b002-e466c99665ea"). InnerVolumeSpecName "kube-api-access-qszqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:18.849118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.849085 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qszqb\" (UniqueName: \"kubernetes.io/projected/f9ec3af2-15dc-4615-b002-e466c99665ea-kube-api-access-qszqb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:18.989173 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.989145 5144 scope.go:122] "RemoveContainer" containerID="ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d" Nov 03 01:00:18.989340 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.989291 5144 scope.go:122] "RemoveContainer" containerID="ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01" Nov 03 01:00:18.989435 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:18.989415 5144 scope.go:122] "RemoveContainer" containerID="8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2" Nov 03 01:00:19.244767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.244740 5144 generic.go:356] "Generic (PLEG): container finished" podID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerID="23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61" exitCode=0 Nov 03 01:00:19.245210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.244815 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" event={"ID":"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803","Type":"ContainerDied","Data":"23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61"} Nov 03 01:00:19.245356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.245327 5144 scope.go:122] "RemoveContainer" containerID="23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61" Nov 03 01:00:19.246846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.246824 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" Nov 03 01:00:19.246985 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.246965 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4" event={"ID":"b1f3731f-2cb5-4e76-8fca-36344dfa207e","Type":"ContainerDied","Data":"0b3d919df710bbd8f8179c30fd15c27f5501f863ff97170057ec0cd5b5eee6c3"} Nov 03 01:00:19.247046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.247032 5144 scope.go:122] "RemoveContainer" containerID="8b200cf2b452fd708fab0c25e10f7331bbd1ea4bfd4002d3f21b5fb60f68006f" Nov 03 01:00:19.247701 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.247671 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:00:19.247851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.247758 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:00:19.247851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.247808 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:00:19.251794 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.251571 5144 generic.go:356] "Generic (PLEG): container finished" podID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerID="078c07aae61c9e50c08d85c89159a57492ef7ae621a2d83974db1266fbe3ad53" exitCode=60 Nov 03 01:00:19.251794 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.251715 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerDied","Data":"078c07aae61c9e50c08d85c89159a57492ef7ae621a2d83974db1266fbe3ad53"} Nov 03 01:00:19.252081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.252065 5144 scope.go:122] "RemoveContainer" containerID="078c07aae61c9e50c08d85c89159a57492ef7ae621a2d83974db1266fbe3ad53" Nov 03 01:00:19.252313 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.252293 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx_odh-ai-gateway-infra(bb8028da-54bb-4a91-9e23-78ba482eafcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" Nov 03 01:00:19.255178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.255156 5144 generic.go:356] "Generic (PLEG): container finished" podID="6241e04a-421b-43e0-aac3-533553ec9b35" containerID="ac3a89ba6265b86f5e40dd8470902fc81b5e5c22924374447725f84183320671" exitCode=60 Nov 03 01:00:19.255299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.255227 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerDied","Data":"ac3a89ba6265b86f5e40dd8470902fc81b5e5c22924374447725f84183320671"} Nov 03 01:00:19.255588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.255550 5144 scope.go:122] "RemoveContainer" containerID="ac3a89ba6265b86f5e40dd8470902fc81b5e5c22924374447725f84183320671" Nov 03 01:00:19.256107 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.256081 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74_odh-ai-gateway-infra(6241e04a-421b-43e0-aac3-533553ec9b35)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" Nov 03 01:00:19.256791 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.256774 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" Nov 03 01:00:19.256791 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.256783 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7" event={"ID":"f9ec3af2-15dc-4615-b002-e466c99665ea","Type":"ContainerDied","Data":"d2366111a9b1178f51e0f24b901478a8c46305fbb3ba909d011e95a214f32416"} Nov 03 01:00:19.258808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.258790 5144 generic.go:356] "Generic (PLEG): container finished" podID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerID="5acc10077003431a2befc32cec8dd234724ed59e66082ea7aff4148b9cec4b2b" exitCode=60 Nov 03 01:00:19.258895 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.258854 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerDied","Data":"5acc10077003431a2befc32cec8dd234724ed59e66082ea7aff4148b9cec4b2b"} Nov 03 01:00:19.259214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.259194 5144 scope.go:122] "RemoveContainer" containerID="5acc10077003431a2befc32cec8dd234724ed59e66082ea7aff4148b9cec4b2b" Nov 03 01:00:19.259418 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.259398 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894460-bvg9f_odh-ai-gateway-infra(de7c42b8-71e6-461d-b8a1-5b76c12b8047)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" Nov 03 01:00:19.262517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.262460 5144 generic.go:356] "Generic (PLEG): container finished" podID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerID="d6fffdab35a0e009ab9447df3c444fa4243e3d9cb5302860fc7a72155968af72" exitCode=60 Nov 03 01:00:19.262610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.262532 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerDied","Data":"d6fffdab35a0e009ab9447df3c444fa4243e3d9cb5302860fc7a72155968af72"} Nov 03 01:00:19.262895 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.262869 5144 scope.go:122] "RemoveContainer" containerID="d6fffdab35a0e009ab9447df3c444fa4243e3d9cb5302860fc7a72155968af72" Nov 03 01:00:19.263102 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.263081 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2_odh-ai-gateway-infra(d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" Nov 03 01:00:19.264282 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.264262 5144 generic.go:356] "Generic (PLEG): container finished" podID="f2257916-7c4d-4956-a42e-32193b32467f" containerID="470fb757b0c551e5e9555a1a3702c5aca44acf7142a11442bdf0ff2fe972caf2" exitCode=60 Nov 03 01:00:19.264369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.264324 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerDied","Data":"470fb757b0c551e5e9555a1a3702c5aca44acf7142a11442bdf0ff2fe972caf2"} Nov 03 01:00:19.264605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.264589 5144 scope.go:122] "RemoveContainer" containerID="470fb757b0c551e5e9555a1a3702c5aca44acf7142a11442bdf0ff2fe972caf2" Nov 03 01:00:19.264840 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.264815 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc_odh-ai-gateway-infra(f2257916-7c4d-4956-a42e-32193b32467f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" podUID="f2257916-7c4d-4956-a42e-32193b32467f" Nov 03 01:00:19.266298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.266277 5144 generic.go:356] "Generic (PLEG): container finished" podID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerID="5849aff0c0e63b13448e40dfb96ed786329ae78b7caf6e2442a5a0426baf1a2d" exitCode=60 Nov 03 01:00:19.266403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.266346 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerDied","Data":"5849aff0c0e63b13448e40dfb96ed786329ae78b7caf6e2442a5a0426baf1a2d"} Nov 03 01:00:19.266742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.266721 5144 scope.go:122] "RemoveContainer" containerID="5849aff0c0e63b13448e40dfb96ed786329ae78b7caf6e2442a5a0426baf1a2d" Nov 03 01:00:19.266974 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.266952 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h_odh-ai-gateway-infra(8c1a1243-df11-4de9-ad33-bce7df581ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" Nov 03 01:00:19.268361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.268337 5144 generic.go:356] "Generic (PLEG): container finished" podID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerID="8c98a7bbbd77d976a1464a4ef99bde37411beb44aced9f7199f670a311d78988" exitCode=60 Nov 03 01:00:19.268439 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.268410 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerDied","Data":"8c98a7bbbd77d976a1464a4ef99bde37411beb44aced9f7199f670a311d78988"} Nov 03 01:00:19.268768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.268750 5144 scope.go:122] "RemoveContainer" containerID="8c98a7bbbd77d976a1464a4ef99bde37411beb44aced9f7199f670a311d78988" Nov 03 01:00:19.268998 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:19.268974 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf_odh-ai-gateway-infra(a022f997-f8e8-4ec5-9438-9b69b291591b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" Nov 03 01:00:19.348968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.346198 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx"] Nov 03 01:00:19.442271 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.442144 5144 scope.go:122] "RemoveContainer" containerID="396ec4bc1ca7eeeb36e645444e28470ed641ff00a9343f5baeed152ec0ced0da" Nov 03 01:00:19.472393 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.472372 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:19.492440 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.492419 5144 scope.go:122] "RemoveContainer" containerID="7ca2107aa88d94ec331662c9003abf72bcefae0b2e22f60bbc93df71b3e56bc5" Nov 03 01:00:19.505447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.505426 5144 scope.go:122] "RemoveContainer" containerID="a1a6db201506272184f9db4a2c5a93f9a9d5ad7cfff3dc516df0b84d13120361" Nov 03 01:00:19.516036 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.516012 5144 scope.go:122] "RemoveContainer" containerID="67112ce7032e059faffb8ebd97de9d7342a18f42e1a5177a303bcecfcd174a3d" Nov 03 01:00:19.536540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.536509 5144 scope.go:122] "RemoveContainer" containerID="77a367b935617c75d4c8e25cc9ac0152dba9133e508d3b292cf17258ccff5524" Nov 03 01:00:19.548754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.548729 5144 scope.go:122] "RemoveContainer" containerID="fa8c185a0e56b3d2dc75a1948ecaca574aadfd0a76ac3ab7ba6fe57fa7148010" Nov 03 01:00:19.562456 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.562428 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca\") pod \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " Nov 03 01:00:19.562583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.562541 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw\" (UniqueName: \"kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw\") pod \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\" (UID: \"b68024ff-ae2e-4ea6-aa70-85a1816be6cd\") " Nov 03 01:00:19.563521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.563500 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca" pod "b68024ff-ae2e-4ea6-aa70-85a1816be6cd" (UID: "b68024ff-ae2e-4ea6-aa70-85a1816be6cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:19.564872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.564853 5144 scope.go:122] "RemoveContainer" containerID="965cdb0cfd9ed8170e58d5e1b276d4ffead1e519ab271c5917eaa0632dc7a9d2" Nov 03 01:00:19.577073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.577055 5144 scope.go:122] "RemoveContainer" containerID="b85785c4d1029c90f0fbbf9a1ba442bf408402022e9c99cffd0c91745c5e8c73" Nov 03 01:00:19.577762 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.577743 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74"] Nov 03 01:00:19.614893 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.614867 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw" pod "b68024ff-ae2e-4ea6-aa70-85a1816be6cd" (UID: "b68024ff-ae2e-4ea6-aa70-85a1816be6cd"). InnerVolumeSpecName "kube-api-access-vvtnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:19.664220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.664184 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.664220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.664212 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtnw\" (UniqueName: \"kubernetes.io/projected/b68024ff-ae2e-4ea6-aa70-85a1816be6cd-kube-api-access-vvtnw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:19.692364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.692330 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2"] Nov 03 01:00:19.806057 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.806016 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf"] Nov 03 01:00:19.988760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:19.988689 5144 scope.go:122] "RemoveContainer" containerID="bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9" Nov 03 01:00:20.278996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.278966 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerStarted","Data":"5e396a7da0cd06fd5108b6e968095abc753fa13899d7bbc10a7ef26f1e719514"} Nov 03 01:00:20.284665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.284622 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" event={"ID":"b68024ff-ae2e-4ea6-aa70-85a1816be6cd","Type":"ContainerDied","Data":"e624aec990b46fccba6f6ccb5d050c03225da15e725e9ca5065f39a6a99cca87"} Nov 03 01:00:20.284766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.284693 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv" Nov 03 01:00:20.284766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.284702 5144 scope.go:122] "RemoveContainer" containerID="474060ed7d056a0fffe116362b53a79c8bb94d0f2ea9a2c90e759d7c7e368ca6" Nov 03 01:00:20.290940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.290918 5144 generic.go:356] "Generic (PLEG): container finished" podID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerID="27e0331663312195fa90873ee67b0b62e92d24ab17cdff91efdc04e5ab5b1fbc" exitCode=60 Nov 03 01:00:20.291043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.290986 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerDied","Data":"27e0331663312195fa90873ee67b0b62e92d24ab17cdff91efdc04e5ab5b1fbc"} Nov 03 01:00:20.291364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.291344 5144 scope.go:122] "RemoveContainer" containerID="27e0331663312195fa90873ee67b0b62e92d24ab17cdff91efdc04e5ab5b1fbc" Nov 03 01:00:20.291665 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:20.291621 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg_odh-ai-gateway-infra(9e5285dd-5ee8-4b87-bbc2-fd9b4870343f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" Nov 03 01:00:20.297081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.297054 5144 generic.go:356] "Generic (PLEG): container finished" podID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerID="21606d448a092c038409bbf475424c55cd3aae087727e26ecf583cd054eb454e" exitCode=60 Nov 03 01:00:20.297204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.297185 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerDied","Data":"21606d448a092c038409bbf475424c55cd3aae087727e26ecf583cd054eb454e"} Nov 03 01:00:20.297548 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.297532 5144 scope.go:122] "RemoveContainer" containerID="21606d448a092c038409bbf475424c55cd3aae087727e26ecf583cd054eb454e" Nov 03 01:00:20.297882 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:20.297858 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c_odh-ai-gateway-infra(62ba0db4-a7a4-455d-a64a-a12879b89dd0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" Nov 03 01:00:20.300088 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.300066 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" event={"ID":"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803","Type":"ContainerStarted","Data":"73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71"} Nov 03 01:00:20.360137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.360112 5144 scope.go:122] "RemoveContainer" containerID="8ec27e7d35d34a6965e6ee9930658990feca5650d02ebe3c091c125e8e8f34c2" Nov 03 01:00:20.406056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.406031 5144 scope.go:122] "RemoveContainer" containerID="ad13ca0eba1e4b8de6c08c5b3c20d3f1e82efee5c4c8327ded3f8fc52d0bbc01" Nov 03 01:00:20.511659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.510728 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4"] Nov 03 01:00:20.524992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.516826 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894460-l5rh4"] Nov 03 01:00:20.543960 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.543875 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:20.575879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.575859 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx\") pod \"6241e04a-421b-43e0-aac3-533553ec9b35\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " Nov 03 01:00:20.578818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.576368 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca\") pod \"6241e04a-421b-43e0-aac3-533553ec9b35\" (UID: \"6241e04a-421b-43e0-aac3-533553ec9b35\") " Nov 03 01:00:20.578818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.577361 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca" pod "6241e04a-421b-43e0-aac3-533553ec9b35" (UID: "6241e04a-421b-43e0-aac3-533553ec9b35"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:20.579105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.579082 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx" pod "6241e04a-421b-43e0-aac3-533553ec9b35" (UID: "6241e04a-421b-43e0-aac3-533553ec9b35"). InnerVolumeSpecName "kube-api-access-pgndx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:20.582463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.582439 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:20.585508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.585485 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:20.590742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.590719 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:20.678336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678301 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca\") pod \"a022f997-f8e8-4ec5-9438-9b69b291591b\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " Nov 03 01:00:20.678537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678380 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2\" (UniqueName: \"kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2\") pod \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " Nov 03 01:00:20.678537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678413 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v\" (UniqueName: \"kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v\") pod \"bb8028da-54bb-4a91-9e23-78ba482eafcd\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " Nov 03 01:00:20.678537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678486 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca\") pod \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\" (UID: \"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429\") " Nov 03 01:00:20.678716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678682 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca\") pod \"bb8028da-54bb-4a91-9e23-78ba482eafcd\" (UID: \"bb8028da-54bb-4a91-9e23-78ba482eafcd\") " Nov 03 01:00:20.678799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678764 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca" pod "a022f997-f8e8-4ec5-9438-9b69b291591b" (UID: "a022f997-f8e8-4ec5-9438-9b69b291591b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:20.678878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678837 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt\" (UniqueName: \"kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt\") pod \"a022f997-f8e8-4ec5-9438-9b69b291591b\" (UID: \"a022f997-f8e8-4ec5-9438-9b69b291591b\") " Nov 03 01:00:20.678990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.678964 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca" pod "d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" (UID: "d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:20.679174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.679150 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca" pod "bb8028da-54bb-4a91-9e23-78ba482eafcd" (UID: "bb8028da-54bb-4a91-9e23-78ba482eafcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:20.679587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.679429 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.679587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.679455 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgndx\" (UniqueName: \"kubernetes.io/projected/6241e04a-421b-43e0-aac3-533553ec9b35-kube-api-access-pgndx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.679587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.679471 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb8028da-54bb-4a91-9e23-78ba482eafcd-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.679587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.679485 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6241e04a-421b-43e0-aac3-533553ec9b35-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.680946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.680921 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v" pod "bb8028da-54bb-4a91-9e23-78ba482eafcd" (UID: "bb8028da-54bb-4a91-9e23-78ba482eafcd"). InnerVolumeSpecName "kube-api-access-dh24v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:20.681119 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.681102 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2" pod "d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" (UID: "d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429"). InnerVolumeSpecName "kube-api-access-lnxm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:20.681276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.681259 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt" pod "a022f997-f8e8-4ec5-9438-9b69b291591b" (UID: "a022f997-f8e8-4ec5-9438-9b69b291591b"). InnerVolumeSpecName "kube-api-access-bhxzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:20.738366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.738334 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 01:00:20.739928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.739908 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 03 01:00:20.740013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.739950 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 03 01:00:20.768901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.768835 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc"] Nov 03 01:00:20.783659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.783513 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxzt\" (UniqueName: \"kubernetes.io/projected/a022f997-f8e8-4ec5-9438-9b69b291591b-kube-api-access-bhxzt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.783659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.783545 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a022f997-f8e8-4ec5-9438-9b69b291591b-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.783659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.783560 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxm2\" (UniqueName: \"kubernetes.io/projected/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429-kube-api-access-lnxm2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.783659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.783574 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh24v\" (UniqueName: \"kubernetes.io/projected/bb8028da-54bb-4a91-9e23-78ba482eafcd-kube-api-access-dh24v\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.928521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.928487 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:20.962654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.958922 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7"] Nov 03 01:00:20.967594 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.967567 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-048135-29894460-s5lf7"] Nov 03 01:00:20.986034 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.986007 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca\") pod \"f2257916-7c4d-4956-a42e-32193b32467f\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " Nov 03 01:00:20.986214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.986200 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr\" (UniqueName: \"kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr\") pod \"f2257916-7c4d-4956-a42e-32193b32467f\" (UID: \"f2257916-7c4d-4956-a42e-32193b32467f\") " Nov 03 01:00:20.986398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.986376 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca" pod "f2257916-7c4d-4956-a42e-32193b32467f" (UID: "f2257916-7c4d-4956-a42e-32193b32467f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:20.986558 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.986546 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2257916-7c4d-4956-a42e-32193b32467f-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:20.988460 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.988442 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr" pod "f2257916-7c4d-4956-a42e-32193b32467f" (UID: "f2257916-7c4d-4956-a42e-32193b32467f"). InnerVolumeSpecName "kube-api-access-mfcvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:20.998361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:20.998329 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h"] Nov 03 01:00:21.087549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.087516 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfcvr\" (UniqueName: \"kubernetes.io/projected/f2257916-7c4d-4956-a42e-32193b32467f-kube-api-access-mfcvr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.110526 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.110501 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f"] Nov 03 01:00:21.286607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.285741 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv"] Nov 03 01:00:21.290695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.290665 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894460-p4rkv"] Nov 03 01:00:21.321650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.318105 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" Nov 03 01:00:21.321650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.318105 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx" event={"ID":"bb8028da-54bb-4a91-9e23-78ba482eafcd","Type":"ContainerDied","Data":"7be957bd91058385b96d57cb20e76de063ee5d9e5dff4663f0554254567e299b"} Nov 03 01:00:21.321650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.318536 5144 scope.go:122] "RemoveContainer" containerID="078c07aae61c9e50c08d85c89159a57492ef7ae621a2d83974db1266fbe3ad53" Nov 03 01:00:21.322716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.322613 5144 generic.go:356] "Generic (PLEG): container finished" podID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerID="97c46141ebe660a7f1c312d96fc591529ab292defadab7f3bdac285fbb8f0d98" exitCode=60 Nov 03 01:00:21.322819 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.322780 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerDied","Data":"97c46141ebe660a7f1c312d96fc591529ab292defadab7f3bdac285fbb8f0d98"} Nov 03 01:00:21.323221 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.323198 5144 scope.go:122] "RemoveContainer" containerID="97c46141ebe660a7f1c312d96fc591529ab292defadab7f3bdac285fbb8f0d98" Nov 03 01:00:21.323565 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:21.323538 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds_odh-ai-gateway-infra(c94d86b5-e3a6-4653-97bc-6456b9a77362)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" Nov 03 01:00:21.326316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.326289 5144 generic.go:356] "Generic (PLEG): container finished" podID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerID="5e396a7da0cd06fd5108b6e968095abc753fa13899d7bbc10a7ef26f1e719514" exitCode=60 Nov 03 01:00:21.326478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.326375 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerDied","Data":"5e396a7da0cd06fd5108b6e968095abc753fa13899d7bbc10a7ef26f1e719514"} Nov 03 01:00:21.326790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.326771 5144 scope.go:122] "RemoveContainer" containerID="5e396a7da0cd06fd5108b6e968095abc753fa13899d7bbc10a7ef26f1e719514" Nov 03 01:00:21.327106 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:21.327081 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s_odh-ai-gateway-infra(e9cd69ff-7d82-4774-bd9a-505db369f298)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" Nov 03 01:00:21.328884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.328858 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" event={"ID":"6241e04a-421b-43e0-aac3-533553ec9b35","Type":"ContainerDied","Data":"825f259607bf84a82a2bcf7d34b4e53f9761287d0676a1c82a2ef231f7c3b6bc"} Nov 03 01:00:21.329009 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.328968 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74" Nov 03 01:00:21.354546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.340647 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" event={"ID":"d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429","Type":"ContainerDied","Data":"9d0b36db7cbc9d9929929141ab361fdb2b46a21e8212dcd2e8fdea1e33d0f802"} Nov 03 01:00:21.354546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.340767 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2" Nov 03 01:00:21.364849 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.364813 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" Nov 03 01:00:21.365472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.365451 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc" event={"ID":"f2257916-7c4d-4956-a42e-32193b32467f","Type":"ContainerDied","Data":"bf3ddaf537f8b50c7c1208c76fa587984f8ce46d4f1feecb4bbde13981830e0f"} Nov 03 01:00:21.372033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.371884 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:21.374073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.372171 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" event={"ID":"8c1a1243-df11-4de9-ad33-bce7df581ac6","Type":"ContainerDied","Data":"17d5f8ff67bbb50e6db117e33a152d9f9294de6f938d00984165ec3ea0e5e7ce"} Nov 03 01:00:21.378618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.378602 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" Nov 03 01:00:21.379227 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.378616 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf" event={"ID":"a022f997-f8e8-4ec5-9438-9b69b291591b","Type":"ContainerDied","Data":"f70f89bdaf35f01880e198a1ec150d50d7c4f73a410d5dc2e28a44ad200035a7"} Nov 03 01:00:21.387550 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.387535 5144 scope.go:122] "RemoveContainer" containerID="bd3e3b5cebe92efa104ced1ae0fedcefaf3aa0dacb05b1684eca0cbc75f6ffc9" Nov 03 01:00:21.391424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.391405 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs\" (UniqueName: \"kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs\") pod \"8c1a1243-df11-4de9-ad33-bce7df581ac6\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " Nov 03 01:00:21.391589 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.391568 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca\") pod \"8c1a1243-df11-4de9-ad33-bce7df581ac6\" (UID: \"8c1a1243-df11-4de9-ad33-bce7df581ac6\") " Nov 03 01:00:21.392040 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.392018 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca" pod "8c1a1243-df11-4de9-ad33-bce7df581ac6" (UID: "8c1a1243-df11-4de9-ad33-bce7df581ac6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.395069 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.395043 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs" pod "8c1a1243-df11-4de9-ad33-bce7df581ac6" (UID: "8c1a1243-df11-4de9-ad33-bce7df581ac6"). InnerVolumeSpecName "kube-api-access-s7wfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.419045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.416609 5144 scope.go:122] "RemoveContainer" containerID="ddccfa05825de054de14368303c4e8d81a531a6a66d6e6cea15126ee35a1751d" Nov 03 01:00:21.460473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.459823 5144 scope.go:122] "RemoveContainer" containerID="ac3a89ba6265b86f5e40dd8470902fc81b5e5c22924374447725f84183320671" Nov 03 01:00:21.484713 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.484690 5144 scope.go:122] "RemoveContainer" containerID="d6fffdab35a0e009ab9447df3c444fa4243e3d9cb5302860fc7a72155968af72" Nov 03 01:00:21.485132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.485108 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:21.493155 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.493101 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7wfs\" (UniqueName: \"kubernetes.io/projected/8c1a1243-df11-4de9-ad33-bce7df581ac6-kube-api-access-s7wfs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.493155 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.493130 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c1a1243-df11-4de9-ad33-bce7df581ac6-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.494642 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.494612 5144 scope.go:122] "RemoveContainer" containerID="470fb757b0c551e5e9555a1a3702c5aca44acf7142a11442bdf0ff2fe972caf2" Nov 03 01:00:21.504269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.504253 5144 scope.go:122] "RemoveContainer" containerID="5849aff0c0e63b13448e40dfb96ed786329ae78b7caf6e2442a5a0426baf1a2d" Nov 03 01:00:21.515086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.515016 5144 scope.go:122] "RemoveContainer" containerID="8c98a7bbbd77d976a1464a4ef99bde37411beb44aced9f7199f670a311d78988" Nov 03 01:00:21.594016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.593943 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp\" (UniqueName: \"kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp\") pod \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " Nov 03 01:00:21.594016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.593994 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca\") pod \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\" (UID: \"de7c42b8-71e6-461d-b8a1-5b76c12b8047\") " Nov 03 01:00:21.594361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.594332 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca" pod "de7c42b8-71e6-461d-b8a1-5b76c12b8047" (UID: "de7c42b8-71e6-461d-b8a1-5b76c12b8047"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:21.596262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.596242 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp" pod "de7c42b8-71e6-461d-b8a1-5b76c12b8047" (UID: "de7c42b8-71e6-461d-b8a1-5b76c12b8047"). InnerVolumeSpecName "kube-api-access-xzjgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:21.696013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.695981 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzjgp\" (UniqueName: \"kubernetes.io/projected/de7c42b8-71e6-461d-b8a1-5b76c12b8047-kube-api-access-xzjgp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.696013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.696007 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de7c42b8-71e6-461d-b8a1-5b76c12b8047-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:21.741109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.741061 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 03 01:00:21.741366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.741339 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 03 01:00:21.997952 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.997875 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" path="/var/lib/kubelet/pods/b1f3731f-2cb5-4e76-8fca-36344dfa207e/volumes" Nov 03 01:00:21.998330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.998315 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" path="/var/lib/kubelet/pods/b68024ff-ae2e-4ea6-aa70-85a1816be6cd/volumes" Nov 03 01:00:21.998746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:21.998731 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" path="/var/lib/kubelet/pods/f9ec3af2-15dc-4615-b002-e466c99665ea/volumes" Nov 03 01:00:22.068518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.068491 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx"] Nov 03 01:00:22.073375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.073352 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6eb9923f-nwnjx"] Nov 03 01:00:22.212806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.212773 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74"] Nov 03 01:00:22.247658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.246120 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894460-ncd74"] Nov 03 01:00:22.406463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.405845 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" Nov 03 01:00:22.406463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.405874 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f" event={"ID":"de7c42b8-71e6-461d-b8a1-5b76c12b8047","Type":"ContainerDied","Data":"05099b2e0ce8a796dd541b3247e3a4b970aae19fe5035f1f1b0d999e4426aa5c"} Nov 03 01:00:22.406463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.405974 5144 scope.go:122] "RemoveContainer" containerID="5acc10077003431a2befc32cec8dd234724ed59e66082ea7aff4148b9cec4b2b" Nov 03 01:00:22.408738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.408707 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg"] Nov 03 01:00:22.435894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.435868 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h" Nov 03 01:00:22.590142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.590110 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx"] Nov 03 01:00:22.610128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.610079 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c19567-29894460-vdnqx"] Nov 03 01:00:22.662700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.661405 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s"] Nov 03 01:00:22.761942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.761236 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 03 01:00:22.761942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.761295 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 03 01:00:22.768785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.766497 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 01:00:22.773320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.773246 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:22.783158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.782537 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c"] Nov 03 01:00:22.811962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.811928 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca\") pod \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " Nov 03 01:00:22.812126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.812020 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp\" (UniqueName: \"kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp\") pod \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\" (UID: \"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f\") " Nov 03 01:00:22.812422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.812378 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca" pod "9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" (UID: "9e5285dd-5ee8-4b87-bbc2-fd9b4870343f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:22.812957 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.812931 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:22.847659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.845913 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp" pod "9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" (UID: "9e5285dd-5ee8-4b87-bbc2-fd9b4870343f"). InnerVolumeSpecName "kube-api-access-2jdvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:22.887832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.887624 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2"] Nov 03 01:00:22.894559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.894533 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-3b1d36-29894460-p7cv2"] Nov 03 01:00:22.898775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.898748 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds"] Nov 03 01:00:22.901051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.901032 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:22.934712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:22.934034 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jdvp\" (UniqueName: \"kubernetes.io/projected/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f-kube-api-access-2jdvp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.053282 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.050691 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb\" (UniqueName: \"kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb\") pod \"e9cd69ff-7d82-4774-bd9a-505db369f298\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " Nov 03 01:00:23.053282 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.050777 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca\") pod \"e9cd69ff-7d82-4774-bd9a-505db369f298\" (UID: \"e9cd69ff-7d82-4774-bd9a-505db369f298\") " Nov 03 01:00:23.056932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.056895 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb" pod "e9cd69ff-7d82-4774-bd9a-505db369f298" (UID: "e9cd69ff-7d82-4774-bd9a-505db369f298"). InnerVolumeSpecName "kube-api-access-x8sqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.057050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.057025 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca" pod "e9cd69ff-7d82-4774-bd9a-505db369f298" (UID: "e9cd69ff-7d82-4774-bd9a-505db369f298"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.059381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.059292 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf"] Nov 03 01:00:23.075921 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.075892 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-904a34-29894460-5rctf"] Nov 03 01:00:23.093724 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.093702 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:23.161656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.154046 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx\" (UniqueName: \"kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx\") pod \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " Nov 03 01:00:23.161656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.154190 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca\") pod \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\" (UID: \"62ba0db4-a7a4-455d-a64a-a12879b89dd0\") " Nov 03 01:00:23.161656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.154781 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8sqb\" (UniqueName: \"kubernetes.io/projected/e9cd69ff-7d82-4774-bd9a-505db369f298-kube-api-access-x8sqb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.161656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.154802 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd69ff-7d82-4774-bd9a-505db369f298-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.161656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.156954 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca" pod "62ba0db4-a7a4-455d-a64a-a12879b89dd0" (UID: "62ba0db4-a7a4-455d-a64a-a12879b89dd0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.161656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.159538 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx" pod "62ba0db4-a7a4-455d-a64a-a12879b89dd0" (UID: "62ba0db4-a7a4-455d-a64a-a12879b89dd0"). InnerVolumeSpecName "kube-api-access-zbzdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.205039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.204982 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:23.257107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.257073 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62ba0db4-a7a4-455d-a64a-a12879b89dd0-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.257107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.257106 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbzdx\" (UniqueName: \"kubernetes.io/projected/62ba0db4-a7a4-455d-a64a-a12879b89dd0-kube-api-access-zbzdx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.260040 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.260006 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc"] Nov 03 01:00:23.266454 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.266428 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894460-fd6hc"] Nov 03 01:00:23.358206 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.358176 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca\") pod \"c94d86b5-e3a6-4653-97bc-6456b9a77362\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " Nov 03 01:00:23.358394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.358281 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9\" (UniqueName: \"kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9\") pod \"c94d86b5-e3a6-4653-97bc-6456b9a77362\" (UID: \"c94d86b5-e3a6-4653-97bc-6456b9a77362\") " Nov 03 01:00:23.358605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.358578 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca" pod "c94d86b5-e3a6-4653-97bc-6456b9a77362" (UID: "c94d86b5-e3a6-4653-97bc-6456b9a77362"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:23.358830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.358803 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c94d86b5-e3a6-4653-97bc-6456b9a77362-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.360683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.360659 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9" pod "c94d86b5-e3a6-4653-97bc-6456b9a77362" (UID: "c94d86b5-e3a6-4653-97bc-6456b9a77362"). InnerVolumeSpecName "kube-api-access-vdcj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:23.455608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.455528 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" Nov 03 01:00:23.455608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.455543 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds" event={"ID":"c94d86b5-e3a6-4653-97bc-6456b9a77362","Type":"ContainerDied","Data":"6a24788c951cf44859c91e697579a284fa0cdf337b35ffc3cb21640f9f2e9a1f"} Nov 03 01:00:23.456164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.455611 5144 scope.go:122] "RemoveContainer" containerID="97c46141ebe660a7f1c312d96fc591529ab292defadab7f3bdac285fbb8f0d98" Nov 03 01:00:23.457690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.457504 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" Nov 03 01:00:23.457690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.457569 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s" event={"ID":"e9cd69ff-7d82-4774-bd9a-505db369f298","Type":"ContainerDied","Data":"df78ea3b93d51fe140f2cd6637ff386210b86fabc60fc702364ec8846ec35e57"} Nov 03 01:00:23.459819 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.459793 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdcj9\" (UniqueName: \"kubernetes.io/projected/c94d86b5-e3a6-4653-97bc-6456b9a77362-kube-api-access-vdcj9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:23.461136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.461113 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" event={"ID":"9e5285dd-5ee8-4b87-bbc2-fd9b4870343f","Type":"ContainerDied","Data":"372b0baac18e9306d587f9a20f3ad178da2c6bd6dd346e2affa0a900f5826cb6"} Nov 03 01:00:23.461217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.461205 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg" Nov 03 01:00:23.468839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.465590 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" Nov 03 01:00:23.468839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.465757 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c" event={"ID":"62ba0db4-a7a4-455d-a64a-a12879b89dd0","Type":"ContainerDied","Data":"fa91b918abbe8008d7afc7554a0fe209fea2c9c77a70e6743dcc31c30559bcd8"} Nov 03 01:00:23.469091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.469077 5144 scope.go:122] "RemoveContainer" containerID="5e396a7da0cd06fd5108b6e968095abc753fa13899d7bbc10a7ef26f1e719514" Nov 03 01:00:23.479866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.479852 5144 scope.go:122] "RemoveContainer" containerID="27e0331663312195fa90873ee67b0b62e92d24ab17cdff91efdc04e5ab5b1fbc" Nov 03 01:00:23.489479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.489451 5144 scope.go:122] "RemoveContainer" containerID="21606d448a092c038409bbf475424c55cd3aae087727e26ecf583cd054eb454e" Nov 03 01:00:23.500080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.500060 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h"] Nov 03 01:00:23.504459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.504439 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-d7bfc2-29894460-m586h"] Nov 03 01:00:23.751703 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.750948 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 03 01:00:23.751703 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.750999 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 03 01:00:23.861711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.861670 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f"] Nov 03 01:00:23.865891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.865848 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894460-bvg9f"] Nov 03 01:00:23.999906 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:23.999869 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" path="/var/lib/kubelet/pods/6241e04a-421b-43e0-aac3-533553ec9b35/volumes" Nov 03 01:00:24.000587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.000571 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" path="/var/lib/kubelet/pods/8c1a1243-df11-4de9-ad33-bce7df581ac6/volumes" Nov 03 01:00:24.001160 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.001141 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" path="/var/lib/kubelet/pods/a022f997-f8e8-4ec5-9438-9b69b291591b/volumes" Nov 03 01:00:24.001589 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.001564 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" path="/var/lib/kubelet/pods/bb8028da-54bb-4a91-9e23-78ba482eafcd/volumes" Nov 03 01:00:24.002384 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.002327 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" path="/var/lib/kubelet/pods/d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429/volumes" Nov 03 01:00:24.002797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.002779 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" path="/var/lib/kubelet/pods/de7c42b8-71e6-461d-b8a1-5b76c12b8047/volumes" Nov 03 01:00:24.003223 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.003204 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb218b76-2d5e-4471-a2ca-d8888cb73b7c" path="/var/lib/kubelet/pods/eb218b76-2d5e-4471-a2ca-d8888cb73b7c/volumes" Nov 03 01:00:24.003915 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.003897 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2257916-7c4d-4956-a42e-32193b32467f" path="/var/lib/kubelet/pods/f2257916-7c4d-4956-a42e-32193b32467f/volumes" Nov 03 01:00:24.211592 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.211563 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg"] Nov 03 01:00:24.246113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.246072 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894460-vvpvg"] Nov 03 01:00:24.346851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.346821 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds"] Nov 03 01:00:24.350916 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.350891 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-c87e2f-29894460-x5tds"] Nov 03 01:00:24.449485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.449447 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s"] Nov 03 01:00:24.454001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.453972 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894460-gp95s"] Nov 03 01:00:24.559790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.559750 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c"] Nov 03 01:00:24.564745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.564719 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894460-zvv9c"] Nov 03 01:00:24.742714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.741957 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 03 01:00:24.742714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:24.742014 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 03 01:00:25.745778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:25.740905 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" start-of-body= Nov 03 01:00:25.745778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:25.740965 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.18:15021/healthz/ready\": dial tcp 10.132.1.18:15021: connect: connection refused" Nov 03 01:00:25.998143 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:25.998066 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" path="/var/lib/kubelet/pods/62ba0db4-a7a4-455d-a64a-a12879b89dd0/volumes" Nov 03 01:00:25.998553 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:25.998538 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" path="/var/lib/kubelet/pods/9e5285dd-5ee8-4b87-bbc2-fd9b4870343f/volumes" Nov 03 01:00:25.998952 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:25.998938 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" path="/var/lib/kubelet/pods/c94d86b5-e3a6-4653-97bc-6456b9a77362/volumes" Nov 03 01:00:25.999697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:25.999659 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" path="/var/lib/kubelet/pods/e9cd69ff-7d82-4774-bd9a-505db369f298/volumes" Nov 03 01:00:26.743886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:26.743856 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 01:00:26.745021 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:26.744997 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 01:00:42.156524 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:42.156471 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:00:52.118420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:52.118386 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz"] Nov 03 01:00:52.231028 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:52.230992 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42"] Nov 03 01:00:55.748911 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:55.748882 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx"] Nov 03 01:00:55.867371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:55.867336 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j"] Nov 03 01:00:57.118798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.118759 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" podUID="2e2102ad-09d9-4e6d-8948-f9b328b85b68" containerName="payload-processing" containerID="cri-o://bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3" gracePeriod=25 Nov 03 01:00:57.231688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.231620 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" podUID="dcb5eb9c-80db-4590-b366-343da3832c2a" containerName="payload-pre-processing" containerID="cri-o://f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4" gracePeriod=25 Nov 03 01:00:57.506655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.481801 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 01:00:57.585202 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.585178 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 01:00:57.605163 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.605134 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz\" (UniqueName: \"kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz\") pod \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " Nov 03 01:00:57.605309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.605178 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume\") pod \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\" (UID: \"2e2102ad-09d9-4e6d-8948-f9b328b85b68\") " Nov 03 01:00:57.605910 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.605879 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume" pod "2e2102ad-09d9-4e6d-8948-f9b328b85b68" (UID: "2e2102ad-09d9-4e6d-8948-f9b328b85b68"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:57.607515 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.607490 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz" pod "2e2102ad-09d9-4e6d-8948-f9b328b85b68" (UID: "2e2102ad-09d9-4e6d-8948-f9b328b85b68"). InnerVolumeSpecName "kube-api-access-njgwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:57.706140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.706083 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume\") pod \"dcb5eb9c-80db-4590-b366-343da3832c2a\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " Nov 03 01:00:57.706254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.706186 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm\" (UniqueName: \"kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm\") pod \"dcb5eb9c-80db-4590-b366-343da3832c2a\" (UID: \"dcb5eb9c-80db-4590-b366-343da3832c2a\") " Nov 03 01:00:57.706489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.706463 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume" pod "dcb5eb9c-80db-4590-b366-343da3832c2a" (UID: "dcb5eb9c-80db-4590-b366-343da3832c2a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:00:57.706546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.706487 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njgwz\" (UniqueName: \"kubernetes.io/projected/2e2102ad-09d9-4e6d-8948-f9b328b85b68-kube-api-access-njgwz\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:57.706546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.706500 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e2102ad-09d9-4e6d-8948-f9b328b85b68-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:57.709024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.708999 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm" pod "dcb5eb9c-80db-4590-b366-343da3832c2a" (UID: "dcb5eb9c-80db-4590-b366-343da3832c2a"). InnerVolumeSpecName "kube-api-access-x29nm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:00:57.793806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.793774 5144 generic.go:356] "Generic (PLEG): container finished" podID="2e2102ad-09d9-4e6d-8948-f9b328b85b68" containerID="bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3" exitCode=0 Nov 03 01:00:57.793999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.793823 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" Nov 03 01:00:57.793999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.793862 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" event={"ID":"2e2102ad-09d9-4e6d-8948-f9b328b85b68","Type":"ContainerDied","Data":"bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3"} Nov 03 01:00:57.793999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.793915 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz" event={"ID":"2e2102ad-09d9-4e6d-8948-f9b328b85b68","Type":"ContainerDied","Data":"a83fdccccf66c91bb8337a38b52cc831f8bd4468da34fb9674b9129551ae283c"} Nov 03 01:00:57.793999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.793944 5144 scope.go:122] "RemoveContainer" containerID="bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3" Nov 03 01:00:57.795543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.795511 5144 generic.go:356] "Generic (PLEG): container finished" podID="dcb5eb9c-80db-4590-b366-343da3832c2a" containerID="f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4" exitCode=0 Nov 03 01:00:57.795698 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.795546 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" event={"ID":"dcb5eb9c-80db-4590-b366-343da3832c2a","Type":"ContainerDied","Data":"f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4"} Nov 03 01:00:57.795698 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.795570 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" Nov 03 01:00:57.795698 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.795609 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42" event={"ID":"dcb5eb9c-80db-4590-b366-343da3832c2a","Type":"ContainerDied","Data":"5d2b9eb77ecb6f36cd0818bb3bd9c8cff755d7639489518cfc172689e457803f"} Nov 03 01:00:57.805981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.805964 5144 scope.go:122] "RemoveContainer" containerID="bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3" Nov 03 01:00:57.806245 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:57.806221 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3\": container with ID starting with bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3 not found: ID does not exist" containerID="bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3" Nov 03 01:00:57.806334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.806251 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3"} err="failed to get container status \"bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3\": rpc error: code = NotFound desc = could not find container \"bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3\": container with ID starting with bf66916e5a43ec5065bf2d562b8a9fc7c59d364a4da927edcd7fa5defc90fad3 not found: ID does not exist" Nov 03 01:00:57.806334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.806270 5144 scope.go:122] "RemoveContainer" containerID="f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4" Nov 03 01:00:57.807429 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.807410 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dcb5eb9c-80db-4590-b366-343da3832c2a-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:57.807490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.807436 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x29nm\" (UniqueName: \"kubernetes.io/projected/dcb5eb9c-80db-4590-b366-343da3832c2a-kube-api-access-x29nm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:00:57.815907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.815891 5144 scope.go:122] "RemoveContainer" containerID="f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4" Nov 03 01:00:57.816160 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:00:57.816143 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4\": container with ID starting with f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4 not found: ID does not exist" containerID="f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4" Nov 03 01:00:57.816205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.816167 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4"} err="failed to get container status \"f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4\": rpc error: code = NotFound desc = could not find container \"f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4\": container with ID starting with f3c0baa97890d14c8065c5e1ca748fd2ad89cb68bd3e859c4f25915af2d603e4 not found: ID does not exist" Nov 03 01:00:57.958141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.958080 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz"] Nov 03 01:00:57.961949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.961931 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-f40c6c-c9db74fb5-pznwz"] Nov 03 01:00:57.998438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:57.998416 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e2102ad-09d9-4e6d-8948-f9b328b85b68" path="/var/lib/kubelet/pods/2e2102ad-09d9-4e6d-8948-f9b328b85b68/volumes" Nov 03 01:00:58.067443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:58.067411 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42"] Nov 03 01:00:58.075002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:58.074982 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-f40c6c-7cb95fb97f-s5p42"] Nov 03 01:00:59.996357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:00:59.996320 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb5eb9c-80db-4590-b366-343da3832c2a" path="/var/lib/kubelet/pods/dcb5eb9c-80db-4590-b366-343da3832c2a/volumes" Nov 03 01:01:00.749778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:00.749737 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" podUID="8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" containerName="payload-processing" containerID="cri-o://b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d" gracePeriod=25 Nov 03 01:01:00.868003 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:00.867944 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" podUID="5802c243-b71a-4217-9775-17e6c6c5008c" containerName="payload-pre-processing" containerID="cri-o://b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55" gracePeriod=25 Nov 03 01:01:01.035473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.035441 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 01:01:01.152714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.150127 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s\" (UniqueName: \"kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s\") pod \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " Nov 03 01:01:01.152714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.150306 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume\") pod \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\" (UID: \"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df\") " Nov 03 01:01:01.154055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.154025 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume" pod "8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" (UID: "8a7df8b4-9939-47a7-8bd9-0a041dfdf3df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:01.162278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.162235 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s" pod "8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" (UID: "8a7df8b4-9939-47a7-8bd9-0a041dfdf3df"). InnerVolumeSpecName "kube-api-access-q6x9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:01.163570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.162554 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 01:01:01.251870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.251836 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:01.251870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.251863 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6x9s\" (UniqueName: \"kubernetes.io/projected/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df-kube-api-access-q6x9s\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:01.353037 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.352951 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq\" (UniqueName: \"kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq\") pod \"5802c243-b71a-4217-9775-17e6c6c5008c\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " Nov 03 01:01:01.353186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.353089 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume\") pod \"5802c243-b71a-4217-9775-17e6c6c5008c\" (UID: \"5802c243-b71a-4217-9775-17e6c6c5008c\") " Nov 03 01:01:01.353458 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.353428 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume" pod "5802c243-b71a-4217-9775-17e6c6c5008c" (UID: "5802c243-b71a-4217-9775-17e6c6c5008c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:01.355227 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.355205 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq" pod "5802c243-b71a-4217-9775-17e6c6c5008c" (UID: "5802c243-b71a-4217-9775-17e6c6c5008c"). InnerVolumeSpecName "kube-api-access-srgcq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:01.454573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.454538 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srgcq\" (UniqueName: \"kubernetes.io/projected/5802c243-b71a-4217-9775-17e6c6c5008c-kube-api-access-srgcq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:01.454573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.454567 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5802c243-b71a-4217-9775-17e6c6c5008c-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:01.837454 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.837406 5144 generic.go:356] "Generic (PLEG): container finished" podID="5802c243-b71a-4217-9775-17e6c6c5008c" containerID="b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55" exitCode=0 Nov 03 01:01:01.837625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.837517 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" Nov 03 01:01:01.837625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.837541 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" event={"ID":"5802c243-b71a-4217-9775-17e6c6c5008c","Type":"ContainerDied","Data":"b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55"} Nov 03 01:01:01.837625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.837598 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j" event={"ID":"5802c243-b71a-4217-9775-17e6c6c5008c","Type":"ContainerDied","Data":"fd1a6ccfa4f6246e1073c5360b94e29094ae8425094e9f522e519925fd2df578"} Nov 03 01:01:01.837833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.837657 5144 scope.go:122] "RemoveContainer" containerID="b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55" Nov 03 01:01:01.840957 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.840924 5144 generic.go:356] "Generic (PLEG): container finished" podID="8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" containerID="b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d" exitCode=0 Nov 03 01:01:01.844681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.841147 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" event={"ID":"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df","Type":"ContainerDied","Data":"b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d"} Nov 03 01:01:01.844681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.841192 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" event={"ID":"8a7df8b4-9939-47a7-8bd9-0a041dfdf3df","Type":"ContainerDied","Data":"79e591c1afaf89bce8059ed3e8f765f15f49c3152b3689e72d6f6119edbdd1ea"} Nov 03 01:01:01.844681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.841262 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx" Nov 03 01:01:01.857933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.857912 5144 scope.go:122] "RemoveContainer" containerID="b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55" Nov 03 01:01:01.858266 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:01.858245 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55\": container with ID starting with b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55 not found: ID does not exist" containerID="b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55" Nov 03 01:01:01.858358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.858275 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55"} err="failed to get container status \"b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55\": rpc error: code = NotFound desc = could not find container \"b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55\": container with ID starting with b36cce3a836befe3906ff0383ed2ccef104ad313f7b5cece13f449fdbcde7f55 not found: ID does not exist" Nov 03 01:01:01.858358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.858296 5144 scope.go:122] "RemoveContainer" containerID="b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d" Nov 03 01:01:01.869087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.869070 5144 scope.go:122] "RemoveContainer" containerID="b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d" Nov 03 01:01:01.869357 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:01.869333 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d\": container with ID starting with b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d not found: ID does not exist" containerID="b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d" Nov 03 01:01:01.869426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:01.869371 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d"} err="failed to get container status \"b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d\": rpc error: code = NotFound desc = could not find container \"b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d\": container with ID starting with b348c889d83d875a82014ccc349c5241b06bc5b5bfca190d4d70bce76569457d not found: ID does not exist" Nov 03 01:01:02.003273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:02.003245 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j"] Nov 03 01:01:02.053710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:02.053623 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-ea50b7-cbf49cd4c-qcb7j"] Nov 03 01:01:02.115538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:02.115506 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx"] Nov 03 01:01:02.119370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:02.119341 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-ea50b7-56b46755cc-pkxsx"] Nov 03 01:01:03.047132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:03.047095 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp"] Nov 03 01:01:03.096081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:03.096053 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f735f37a-rqjcp"] Nov 03 01:01:03.996782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:03.996745 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5802c243-b71a-4217-9775-17e6c6c5008c" path="/var/lib/kubelet/pods/5802c243-b71a-4217-9775-17e6c6c5008c/volumes" Nov 03 01:01:03.997159 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:03.997145 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" path="/var/lib/kubelet/pods/8a7df8b4-9939-47a7-8bd9-0a041dfdf3df/volumes" Nov 03 01:01:03.997530 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:03.997517 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf2c13a-5ca2-4e72-a229-bb4747fef695" path="/var/lib/kubelet/pods/caf2c13a-5ca2-4e72-a229-bb4747fef695/volumes" Nov 03 01:01:12.157474 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:12.157428 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:01:12.917087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:12.917055 5144 scope.go:122] "RemoveContainer" containerID="564f6860f93c9baa4c1791f83dea200a576fe325b33f53d6126af0e63c4091c1" Nov 03 01:01:14.050314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:14.050266 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq"] Nov 03 01:01:14.101235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:14.101199 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3bb471c-q82fq"] Nov 03 01:01:15.373766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:15.373722 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7"] Nov 03 01:01:15.997842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:15.997811 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc6852e6-ff27-4e7c-ac40-0399e7455cde" path="/var/lib/kubelet/pods/bc6852e6-ff27-4e7c-ac40-0399e7455cde/volumes" Nov 03 01:01:16.584747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.584710 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j"] Nov 03 01:01:16.585381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585366 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585383 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585399 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585403 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585411 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585415 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585421 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585425 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585432 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585438 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585448 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dcb5eb9c-80db-4590-b366-343da3832c2a" containerName="payload-pre-processing" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585453 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb5eb9c-80db-4590-b366-343da3832c2a" containerName="payload-pre-processing" Nov 03 01:01:16.585455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585459 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585463 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585471 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585475 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585480 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585484 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585490 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585495 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585500 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585504 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585511 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585515 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585520 5144 cpu_manager.go:414] "containerMap: removing container" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585524 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585531 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585535 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585541 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585544 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585551 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585555 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585563 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585568 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585573 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585578 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585583 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585587 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585592 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585596 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585604 5144 cpu_manager.go:414] "containerMap: removing container" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585608 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585614 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585618 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585624 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5802c243-b71a-4217-9775-17e6c6c5008c" containerName="payload-pre-processing" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585644 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5802c243-b71a-4217-9775-17e6c6c5008c" containerName="payload-pre-processing" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585652 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585656 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585662 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585666 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585675 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585678 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585684 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585688 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585699 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585703 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585714 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585719 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585724 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585729 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585737 5144 cpu_manager.go:414] "containerMap: removing container" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585741 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585751 5144 cpu_manager.go:414] "containerMap: removing container" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585756 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585767 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2e2102ad-09d9-4e6d-8948-f9b328b85b68" containerName="payload-processing" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585772 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2102ad-09d9-4e6d-8948-f9b328b85b68" containerName="payload-processing" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585780 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" containerName="payload-processing" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585785 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" containerName="payload-processing" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585793 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.585826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585798 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585923 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585932 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585938 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585944 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585952 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585957 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585963 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585969 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585976 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585981 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2102ad-09d9-4e6d-8948-f9b328b85b68" containerName="payload-processing" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585986 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585991 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a7df8b4-9939-47a7-8bd9-0a041dfdf3df" containerName="payload-processing" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.585998 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586003 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586009 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586016 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586021 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586026 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586031 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586036 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586042 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb5eb9c-80db-4590-b366-343da3832c2a" containerName="payload-pre-processing" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586048 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586052 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94d86b5-e3a6-4653-97bc-6456b9a77362" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586057 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586062 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586067 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586072 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586079 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586083 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586089 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586094 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586099 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586104 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586109 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5802c243-b71a-4217-9775-17e6c6c5008c" containerName="payload-pre-processing" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586116 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a022f997-f8e8-4ec5-9438-9b69b291591b" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586123 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:16.587904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.586128 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:16.591951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.591929 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.643872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.643833 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:01:16.644048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.644006 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:01:16.762153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.762090 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.762574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.762519 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.762861 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.762803 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwsk5\" (UniqueName: \"kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.784803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.784769 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j"] Nov 03 01:01:16.890713 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.890444 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwsk5\" (UniqueName: \"kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.890907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.890708 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.890907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.890769 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.906664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.892305 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.932652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.928579 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:16.950688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:16.950622 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwsk5\" (UniqueName: \"kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5\") pod \"maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:17.206153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:17.206071 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:17.610546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:17.610498 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j"] Nov 03 01:01:17.610990 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:01:17.610754 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc78e05a_8fff_41c7_bbb5_b886c6167041.slice/crio-c882ad71f880b0c9196c46b4d4f6c1a7dafd1389708883fe998351e9263b06f9 WatchSource:0}: Error finding container c882ad71f880b0c9196c46b4d4f6c1a7dafd1389708883fe998351e9263b06f9: Status 404 returned error can't find the container with id c882ad71f880b0c9196c46b4d4f6c1a7dafd1389708883fe998351e9263b06f9 Nov 03 01:01:17.976815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:17.976775 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" event={"ID":"fc78e05a-8fff-41c7-bbb5-b886c6167041","Type":"ContainerStarted","Data":"3e270989d59701a626fd610aee0dc4f5078a55ff520c6696ca436ee8b477fda6"} Nov 03 01:01:17.977195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:17.976837 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" event={"ID":"fc78e05a-8fff-41c7-bbb5-b886c6167041","Type":"ContainerStarted","Data":"c882ad71f880b0c9196c46b4d4f6c1a7dafd1389708883fe998351e9263b06f9"} Nov 03 01:01:18.092263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:18.092193 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" podStartSLOduration=2.092173065 podStartE2EDuration="2.092173065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:18.088959493 +0000 UTC m=+3368.234653976" watchObservedRunningTime="2026-11-03 01:01:18.092173065 +0000 UTC m=+3368.237867521" Nov 03 01:01:18.800617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:18.800530 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4"] Nov 03 01:01:18.801109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:18.800793 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" containerID="cri-o://0920dfa2603b50b5edf4c4193ecb464d3fde40239c087d01916062bf4a885adc" gracePeriod=30 Nov 03 01:01:18.993096 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:18.993018 5144 generic.go:356] "Generic (PLEG): container finished" podID="fc78e05a-8fff-41c7-bbb5-b886c6167041" containerID="3e270989d59701a626fd610aee0dc4f5078a55ff520c6696ca436ee8b477fda6" exitCode=0 Nov 03 01:01:18.993253 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:18.993103 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" event={"ID":"fc78e05a-8fff-41c7-bbb5-b886c6167041","Type":"ContainerDied","Data":"3e270989d59701a626fd610aee0dc4f5078a55ff520c6696ca436ee8b477fda6"} Nov 03 01:01:20.182118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.182090 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:20.340956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.340923 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5\" (UniqueName: \"kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5\") pod \"fc78e05a-8fff-41c7-bbb5-b886c6167041\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " Nov 03 01:01:20.341158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.340989 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca\") pod \"fc78e05a-8fff-41c7-bbb5-b886c6167041\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " Nov 03 01:01:20.341158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.341075 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca\") pod \"fc78e05a-8fff-41c7-bbb5-b886c6167041\" (UID: \"fc78e05a-8fff-41c7-bbb5-b886c6167041\") " Nov 03 01:01:20.341492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.341464 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca" pod "fc78e05a-8fff-41c7-bbb5-b886c6167041" (UID: "fc78e05a-8fff-41c7-bbb5-b886c6167041"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:20.341681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.341661 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:20.343433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.343403 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca" pod "fc78e05a-8fff-41c7-bbb5-b886c6167041" (UID: "fc78e05a-8fff-41c7-bbb5-b886c6167041"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:01:20.343538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.343454 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5" pod "fc78e05a-8fff-41c7-bbb5-b886c6167041" (UID: "fc78e05a-8fff-41c7-bbb5-b886c6167041"). InnerVolumeSpecName "kube-api-access-xwsk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:20.374919 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.374878 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" podUID="2c9ed56a-6442-42c5-b222-576f0f0cf687" containerName="payload-pre-processing" containerID="cri-o://a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844" gracePeriod=25 Nov 03 01:01:20.442592 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.442549 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwsk5\" (UniqueName: \"kubernetes.io/projected/fc78e05a-8fff-41c7-bbb5-b886c6167041-kube-api-access-xwsk5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:20.442592 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.442590 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc78e05a-8fff-41c7-bbb5-b886c6167041-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:20.651714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.651674 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 01:01:20.746284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.746249 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume\") pod \"2c9ed56a-6442-42c5-b222-576f0f0cf687\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " Nov 03 01:01:20.746507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.746471 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml\" (UniqueName: \"kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml\") pod \"2c9ed56a-6442-42c5-b222-576f0f0cf687\" (UID: \"2c9ed56a-6442-42c5-b222-576f0f0cf687\") " Nov 03 01:01:20.746759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.746733 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume" pod "2c9ed56a-6442-42c5-b222-576f0f0cf687" (UID: "2c9ed56a-6442-42c5-b222-576f0f0cf687"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:20.746958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.746940 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2c9ed56a-6442-42c5-b222-576f0f0cf687-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:20.748788 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.748767 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml" pod "2c9ed56a-6442-42c5-b222-576f0f0cf687" (UID: "2c9ed56a-6442-42c5-b222-576f0f0cf687"). InnerVolumeSpecName "kube-api-access-wmrml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:20.848386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:20.848350 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrml\" (UniqueName: \"kubernetes.io/projected/2c9ed56a-6442-42c5-b222-576f0f0cf687-kube-api-access-wmrml\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:21.011549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.011496 5144 generic.go:356] "Generic (PLEG): container finished" podID="2c9ed56a-6442-42c5-b222-576f0f0cf687" containerID="a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844" exitCode=0 Nov 03 01:01:21.011801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.011549 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" Nov 03 01:01:21.011801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.011549 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" event={"ID":"2c9ed56a-6442-42c5-b222-576f0f0cf687","Type":"ContainerDied","Data":"a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844"} Nov 03 01:01:21.011801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.011718 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7" event={"ID":"2c9ed56a-6442-42c5-b222-576f0f0cf687","Type":"ContainerDied","Data":"b7c182b86160cd9778be35719e99473508543c3ab138be485e9caac4c70d9223"} Nov 03 01:01:21.011801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.011750 5144 scope.go:122] "RemoveContainer" containerID="a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844" Nov 03 01:01:21.014296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.014275 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" Nov 03 01:01:21.014447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.014379 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j" event={"ID":"fc78e05a-8fff-41c7-bbb5-b886c6167041","Type":"ContainerDied","Data":"c882ad71f880b0c9196c46b4d4f6c1a7dafd1389708883fe998351e9263b06f9"} Nov 03 01:01:21.014447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.014428 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c882ad71f880b0c9196c46b4d4f6c1a7dafd1389708883fe998351e9263b06f9" Nov 03 01:01:21.025668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.025642 5144 scope.go:122] "RemoveContainer" containerID="a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844" Nov 03 01:01:21.025997 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:21.025977 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844\": container with ID starting with a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844 not found: ID does not exist" containerID="a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844" Nov 03 01:01:21.026067 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.026007 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844"} err="failed to get container status \"a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844\": rpc error: code = NotFound desc = could not find container \"a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844\": container with ID starting with a58fa687d8b4df4349637abdd0021b2ea310353145394511e681a914e61f0844 not found: ID does not exist" Nov 03 01:01:21.216457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.216423 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89"] Nov 03 01:01:21.217383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217366 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:21.217445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217386 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb8028da-54bb-4a91-9e23-78ba482eafcd" containerName="cleanup" Nov 03 01:01:21.217445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217406 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:21.217445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217414 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:21.217445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217435 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:21.217445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217443 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217456 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217464 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217472 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217479 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217492 5144 cpu_manager.go:414] "containerMap: removing container" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217498 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217507 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217514 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217525 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2c9ed56a-6442-42c5-b222-576f0f0cf687" containerName="payload-pre-processing" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217531 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9ed56a-6442-42c5-b222-576f0f0cf687" containerName="payload-pre-processing" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217547 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217553 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217590 5144 cpu_manager.go:414] "containerMap: removing container" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217596 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62ba0db4-a7a4-455d-a64a-a12879b89dd0" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217619 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217626 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68024ff-ae2e-4ea6-aa70-85a1816be6cd" containerName="cleanup" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217655 5144 cpu_manager.go:414] "containerMap: removing container" podUID="fc78e05a-8fff-41c7-bbb5-b886c6167041" containerName="revoke-keys" Nov 03 01:01:21.217662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217662 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc78e05a-8fff-41c7-bbb5-b886c6167041" containerName="revoke-keys" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217692 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217698 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217832 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6241e04a-421b-43e0-aac3-533553ec9b35" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217842 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9ec3af2-15dc-4615-b002-e466c99665ea" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217856 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2257916-7c4d-4956-a42e-32193b32467f" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217866 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e5285dd-5ee8-4b87-bbc2-fd9b4870343f" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217875 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cd69ff-7d82-4774-bd9a-505db369f298" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217883 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9ed56a-6442-42c5-b222-576f0f0cf687" containerName="payload-pre-processing" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217891 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1f3731f-2cb5-4e76-8fca-36344dfa207e" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217900 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c1a1243-df11-4de9-ad33-bce7df581ac6" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217909 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7c42b8-71e6-461d-b8a1-5b76c12b8047" containerName="cleanup" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217921 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc78e05a-8fff-41c7-bbb5-b886c6167041" containerName="revoke-keys" Nov 03 01:01:21.218275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.217934 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6ff3a32-0ca5-4a51-9c79-8d4e9afb4429" containerName="cleanup" Nov 03 01:01:21.223434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.223412 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.229279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.229247 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7"] Nov 03 01:01:21.235140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.235116 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-087f6c-66c8fd6df9-b8zl7"] Nov 03 01:01:21.347719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.346001 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89"] Nov 03 01:01:21.347719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.347099 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-cac9f3-openshift-default-dockercfg-46qwz\"" Nov 03 01:01:21.350428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.350109 5144 patch_prober.go:29] interesting pod/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:01:21.350428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.350161 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:01:21.387681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.387625 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.388010 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.387985 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.388186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.388168 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.388310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.388294 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.388450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.388433 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66rt\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.388749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.388730 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.388964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.388947 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.389147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.389128 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.389583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.389560 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.490702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.490667 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.490934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.490726 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.490934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.490835 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.490934 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.490994 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491037 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491064 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491110 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66rt\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491138 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491180 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491446 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491402 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491646 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491739 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.491902 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.491810 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.493402 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.493375 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.494255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.494233 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.570463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.570417 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66rt\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.574691 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.572182 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token\") pod \"e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.838880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.838838 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:21.997295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:21.997256 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9ed56a-6442-42c5-b222-576f0f0cf687" path="/var/lib/kubelet/pods/2c9ed56a-6442-42c5-b222-576f0f0cf687/volumes" Nov 03 01:01:22.274465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:22.274428 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89"] Nov 03 01:01:22.274934 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:01:22.274619 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9006f61a_46ec_4dbb_94b0_afca485bd4e2.slice/crio-70d365c11b5ae638dcd95d2a6c395416a30d5857bbfb0a3f7eccbef045616ae7 WatchSource:0}: Error finding container 70d365c11b5ae638dcd95d2a6c395416a30d5857bbfb0a3f7eccbef045616ae7: Status 404 returned error can't find the container with id 70d365c11b5ae638dcd95d2a6c395416a30d5857bbfb0a3f7eccbef045616ae7 Nov 03 01:01:22.278038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:22.277996 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:01:22.278132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:22.278088 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:01:22.278180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:22.278135 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:01:23.032317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:23.032274 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" event={"ID":"9006f61a-46ec-4dbb-94b0-afca485bd4e2","Type":"ContainerStarted","Data":"bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3"} Nov 03 01:01:23.032517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:23.032343 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" event={"ID":"9006f61a-46ec-4dbb-94b0-afca485bd4e2","Type":"ContainerStarted","Data":"70d365c11b5ae638dcd95d2a6c395416a30d5857bbfb0a3f7eccbef045616ae7"} Nov 03 01:01:23.203397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:23.203341 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podStartSLOduration=2.20332443 podStartE2EDuration="2.20332443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:23.199844901 +0000 UTC m=+3373.345539362" watchObservedRunningTime="2026-11-03 01:01:23.20332443 +0000 UTC m=+3373.349018890" Nov 03 01:01:23.840040 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:23.840004 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:23.846382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:23.846341 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:24.053401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.051674 5144 generic.go:356] "Generic (PLEG): container finished" podID="ddceed9b-f049-40aa-9db7-246a770583d4" containerID="0920dfa2603b50b5edf4c4193ecb464d3fde40239c087d01916062bf4a885adc" exitCode=0 Nov 03 01:01:24.053401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.053004 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" event={"ID":"ddceed9b-f049-40aa-9db7-246a770583d4","Type":"ContainerDied","Data":"0920dfa2603b50b5edf4c4193ecb464d3fde40239c087d01916062bf4a885adc"} Nov 03 01:01:24.053401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.053075 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:24.053401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.053109 5144 scope.go:122] "RemoveContainer" containerID="e8239d2dafdfaf06e919bcad0188df012baac4ef850163254ded98533a676106" Nov 03 01:01:24.054880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.054229 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:01:24.150742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.150708 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 01:01:24.232185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232127 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232198 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232233 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232305 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232368 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232421 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232476 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232613 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.232786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.232675 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data\") pod \"ddceed9b-f049-40aa-9db7-246a770583d4\" (UID: \"ddceed9b-f049-40aa-9db7-246a770583d4\") " Nov 03 01:01:24.233779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.233520 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:24.234868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.234601 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:24.234868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.234775 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:24.235731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.235703 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:24.235941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.235903 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:24.237859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.237431 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:24.237859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.237487 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "kube-api-access-z5d8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:24.237859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.237715 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:24.240454 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.239983 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo" pod "ddceed9b-f049-40aa-9db7-246a770583d4" (UID: "ddceed9b-f049-40aa-9db7-246a770583d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:24.334818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334711 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.334818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334750 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.334818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334783 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddceed9b-f049-40aa-9db7-246a770583d4-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.334818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334797 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.334818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334811 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.334818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334825 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.335426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334839 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddceed9b-f049-40aa-9db7-246a770583d4-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.335426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334853 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddceed9b-f049-40aa-9db7-246a770583d4-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:24.335426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:24.334869 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5d8k\" (UniqueName: \"kubernetes.io/projected/ddceed9b-f049-40aa-9db7-246a770583d4-kube-api-access-z5d8k\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:25.061994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:25.061967 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" Nov 03 01:01:25.061994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:25.061988 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4" event={"ID":"ddceed9b-f049-40aa-9db7-246a770583d4","Type":"ContainerDied","Data":"17680345732cdd34a50197b3498f94e3d3f8a473fd29dca5187b0dbbefb10969"} Nov 03 01:01:25.062580 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:25.062048 5144 scope.go:122] "RemoveContainer" containerID="0920dfa2603b50b5edf4c4193ecb464d3fde40239c087d01916062bf4a885adc" Nov 03 01:01:25.227617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:25.227582 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4"] Nov 03 01:01:25.277563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:25.277522 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-ea50b7-openshift-default-7589c89f95-vpjf4"] Nov 03 01:01:26.006954 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:26.006909 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" path="/var/lib/kubelet/pods/ddceed9b-f049-40aa-9db7-246a770583d4/volumes" Nov 03 01:01:29.843850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.843810 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg"] Nov 03 01:01:29.844872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.844849 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" Nov 03 01:01:29.844932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.844875 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" Nov 03 01:01:29.844980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.844949 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" Nov 03 01:01:29.844980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.844957 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" Nov 03 01:01:29.845140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.845128 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" Nov 03 01:01:29.850925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.850900 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:29.904784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.904750 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg"] Nov 03 01:01:29.957478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.948947 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cac9f3\"" Nov 03 01:01:29.957478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:29.949191 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cac9f3-dockercfg-f86dh\"" Nov 03 01:01:30.039656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.039077 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.039656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.039268 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7ffq\" (UniqueName: \"kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq\") pod \"payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.039656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.039453 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp"] Nov 03 01:01:30.046724 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.042120 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddceed9b-f049-40aa-9db7-246a770583d4" containerName="istio-proxy" Nov 03 01:01:30.051837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.051800 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.067923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.067897 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp"] Nov 03 01:01:30.140869 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.140758 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdvhb\" (UniqueName: \"kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb\") pod \"payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.141054 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.140958 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7ffq\" (UniqueName: \"kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq\") pod \"payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.141105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.141061 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.141105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.141086 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.141797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.141774 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.200213 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.200180 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7ffq\" (UniqueName: \"kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq\") pod \"payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.242826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.242771 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.243068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.242887 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdvhb\" (UniqueName: \"kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb\") pod \"payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.243545 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.243523 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.299617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.299574 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdvhb\" (UniqueName: \"kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb\") pod \"payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.377513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.377476 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:30.470510 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.470337 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:30.769016 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:01:30.768971 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4ffa9a8_3e55_4b34_bd4b_fb133117a2f6.slice/crio-eb0656ff9304cf3b1b413b0d796e58c52839f01010051fd2a716242374ddc8a5 WatchSource:0}: Error finding container eb0656ff9304cf3b1b413b0d796e58c52839f01010051fd2a716242374ddc8a5: Status 404 returned error can't find the container with id eb0656ff9304cf3b1b413b0d796e58c52839f01010051fd2a716242374ddc8a5 Nov 03 01:01:30.769311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.769287 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp"] Nov 03 01:01:30.869771 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:01:30.869738 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86bfd71_23b3_43a4_aecc_306e7edcd7d3.slice/crio-c657e28b160c5ae6fb0fa3687a559f8b15ae1f9d0c67dafd7545d2c197759001 WatchSource:0}: Error finding container c657e28b160c5ae6fb0fa3687a559f8b15ae1f9d0c67dafd7545d2c197759001: Status 404 returned error can't find the container with id c657e28b160c5ae6fb0fa3687a559f8b15ae1f9d0c67dafd7545d2c197759001 Nov 03 01:01:30.870129 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:30.869807 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg"] Nov 03 01:01:31.113246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.113146 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" event={"ID":"e86bfd71-23b3-43a4-aecc-306e7edcd7d3","Type":"ContainerStarted","Data":"67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c"} Nov 03 01:01:31.113246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.113216 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:31.113246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.113237 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" event={"ID":"e86bfd71-23b3-43a4-aecc-306e7edcd7d3","Type":"ContainerStarted","Data":"c657e28b160c5ae6fb0fa3687a559f8b15ae1f9d0c67dafd7545d2c197759001"} Nov 03 01:01:31.115262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.115231 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" event={"ID":"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6","Type":"ContainerStarted","Data":"bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24"} Nov 03 01:01:31.115420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.115289 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" event={"ID":"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6","Type":"ContainerStarted","Data":"eb0656ff9304cf3b1b413b0d796e58c52839f01010051fd2a716242374ddc8a5"} Nov 03 01:01:31.115420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.115400 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:31.230111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.230049 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" podStartSLOduration=2.230033139 podStartE2EDuration="2.230033139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:31.225501021 +0000 UTC m=+3381.371195479" watchObservedRunningTime="2026-11-03 01:01:31.230033139 +0000 UTC m=+3381.375727590" Nov 03 01:01:31.346453 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:31.346286 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" podStartSLOduration=2.346257486 podStartE2EDuration="2.346257486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:31.338413158 +0000 UTC m=+3381.484107639" watchObservedRunningTime="2026-11-03 01:01:31.346257486 +0000 UTC m=+3381.491951945" Nov 03 01:01:37.371844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:37.371793 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv"] Nov 03 01:01:37.372388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:37.372129 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" containerID="cri-o://73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71" gracePeriod=30 Nov 03 01:01:41.747259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:41.747215 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:01:41.747740 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:41.747278 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:01:42.127505 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.127460 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:01:42.127746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.127555 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:01:42.163681 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:42.162572 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:01:42.674400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.674347 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 01:01:42.746112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.745778 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.745857 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.745938 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.745996 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.746144 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.746183 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.746237 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.746516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.746340 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.751577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.746774 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs\") pod \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\" (UID: \"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803\") " Nov 03 01:01:42.751577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.747521 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:42.751577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.747683 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:42.751577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.748063 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:42.751577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.749285 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:01:42.751577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.749486 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:42.764078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.752446 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:01:42.764078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.753035 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "kube-api-access-sn7qz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:42.764078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.755582 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:01:42.764078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.755799 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token" pod "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" (UID: "4c6dc9be-ef7c-4aa7-aa6d-238dc7245803"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:01:42.848482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848430 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848483 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848499 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848514 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848530 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848544 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848559 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn7qz\" (UniqueName: \"kubernetes.io/projected/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-kube-api-access-sn7qz\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848574 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:42.848831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:42.848590 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:01:43.229269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.229222 5144 generic.go:356] "Generic (PLEG): container finished" podID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerID="73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71" exitCode=0 Nov 03 01:01:43.229495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.229369 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" Nov 03 01:01:43.229495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.229379 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" event={"ID":"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803","Type":"ContainerDied","Data":"73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71"} Nov 03 01:01:43.229495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.229451 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv" event={"ID":"4c6dc9be-ef7c-4aa7-aa6d-238dc7245803","Type":"ContainerDied","Data":"47af7fc52e32b4326e72c175d26f49b658a85a5349ca639e1567ac20f1aa1085"} Nov 03 01:01:43.229737 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.229492 5144 scope.go:122] "RemoveContainer" containerID="73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71" Nov 03 01:01:43.243233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.243208 5144 scope.go:122] "RemoveContainer" containerID="23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61" Nov 03 01:01:43.257157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.257132 5144 scope.go:122] "RemoveContainer" containerID="73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71" Nov 03 01:01:43.257597 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:43.257576 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71\": container with ID starting with 73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71 not found: ID does not exist" containerID="73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71" Nov 03 01:01:43.257689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.257608 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71"} err="failed to get container status \"73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71\": rpc error: code = NotFound desc = could not find container \"73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71\": container with ID starting with 73dee9555de5c7dc3910e4920285e6145a3a1890459b17715b752da069a3ed71 not found: ID does not exist" Nov 03 01:01:43.257689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.257646 5144 scope.go:122] "RemoveContainer" containerID="23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61" Nov 03 01:01:43.257955 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:01:43.257936 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61\": container with ID starting with 23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61 not found: ID does not exist" containerID="23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61" Nov 03 01:01:43.258002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.257959 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61"} err="failed to get container status \"23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61\": rpc error: code = NotFound desc = could not find container \"23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61\": container with ID starting with 23910fcf617121d0885baaed784b44736079fd007c34472d467bddd00e0e4d61 not found: ID does not exist" Nov 03 01:01:43.398934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.398887 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv"] Nov 03 01:01:43.448671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.448611 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-087f6c-openshift-default-5b78db6ddf-6mwrv"] Nov 03 01:01:43.998341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:43.998291 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" path="/var/lib/kubelet/pods/4c6dc9be-ef7c-4aa7-aa6d-238dc7245803/volumes" Nov 03 01:01:44.852853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.852804 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-546ff98758-n7jm4"] Nov 03 01:01:44.853940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.853915 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" Nov 03 01:01:44.854000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.853943 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" Nov 03 01:01:44.854043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.854011 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" Nov 03 01:01:44.854043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.854019 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" Nov 03 01:01:44.854183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.854170 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" Nov 03 01:01:44.860115 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.860086 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:44.910725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.910685 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-546ff98758-n7jm4"] Nov 03 01:01:44.934199 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.934160 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9b79b389f9e6\"" Nov 03 01:01:44.934533 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.934511 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:01:44.934680 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.934535 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:01:44.934680 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.934601 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:01:44.934680 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.934605 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:01:44.978261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.978220 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-src7j\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:44.978456 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.978369 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:44.978456 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.978445 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:44.978551 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:44.978527 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.080023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.079974 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-src7j\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.080555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.080066 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.080555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.080175 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.080555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.080259 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.080980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.080951 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.081102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.081080 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.083185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.083162 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.141038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.140939 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-src7j\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j\") pod \"payload-processing-external-model-546ff98758-n7jm4\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.176043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.175998 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:45.565087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:45.565046 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-546ff98758-n7jm4"] Nov 03 01:01:45.566191 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:01:45.566156 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1379a12e_6e59_4529_b42c_c961ff42c78c.slice/crio-f6d845227f575487e8f4184d10579f33ec61026bd3a57f175ea677b27b701531 WatchSource:0}: Error finding container f6d845227f575487e8f4184d10579f33ec61026bd3a57f175ea677b27b701531: Status 404 returned error can't find the container with id f6d845227f575487e8f4184d10579f33ec61026bd3a57f175ea677b27b701531 Nov 03 01:01:46.291947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:46.291908 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" event={"ID":"1379a12e-6e59-4529-b42c-c961ff42c78c","Type":"ContainerStarted","Data":"764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971"} Nov 03 01:01:46.292386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:46.291962 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" event={"ID":"1379a12e-6e59-4529-b42c-c961ff42c78c","Type":"ContainerStarted","Data":"f6d845227f575487e8f4184d10579f33ec61026bd3a57f175ea677b27b701531"} Nov 03 01:01:46.292386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:46.292048 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:46.406948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:46.406887 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" podStartSLOduration=2.406870862 podStartE2EDuration="2.406870862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:01:46.406003817 +0000 UTC m=+3396.551698294" watchObservedRunningTime="2026-11-03 01:01:46.406870862 +0000 UTC m=+3396.552565359" Nov 03 01:01:48.045828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:48.045786 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46"] Nov 03 01:01:48.095173 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:48.095137 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-039f22b6-nhj46"] Nov 03 01:01:49.998488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:49.998440 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5bdc82-0cc5-4278-b4aa-07a45d76ce34" path="/var/lib/kubelet/pods/6a5bdc82-0cc5-4278-b4aa-07a45d76ce34/volumes" Nov 03 01:01:52.046820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:52.046780 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q"] Nov 03 01:01:52.096647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:52.096598 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-836ef7f5-7dp4q"] Nov 03 01:01:53.997202 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:53.997164 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbffd7e9-cc44-4ee1-b6fa-4693fae3b501" path="/var/lib/kubelet/pods/cbffd7e9-cc44-4ee1-b6fa-4693fae3b501/volumes" Nov 03 01:01:55.354892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:55.354852 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p"] Nov 03 01:01:55.355415 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:55.355125 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" containerID="cri-o://48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c" gracePeriod=30 Nov 03 01:01:57.309534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:57.309484 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:01:59.048752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:59.048710 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr"] Nov 03 01:01:59.053008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:59.052977 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:01:59.053176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:59.053034 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:01:59.096318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:59.096276 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-441ee790-6q5wr"] Nov 03 01:01:59.997268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:01:59.997230 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf6c5d92-02ed-45ec-894d-82ce24703fe1" path="/var/lib/kubelet/pods/bf6c5d92-02ed-45ec-894d-82ce24703fe1/volumes" Nov 03 01:02:00.642005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.641927 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 01:02:00.745997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.745953 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746008 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746157 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746198 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746233 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746361 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746400 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746594 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746480 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746594 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746531 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs\") pod \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\" (UID: \"51b09322-4337-4ff7-b0e4-bb3cc82436d5\") " Nov 03 01:02:00.746766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746603 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.746766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746621 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:00.746877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746842 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.746951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.746924 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.747423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.747239 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.747423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.747356 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.747423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.747372 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.747423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.747387 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.747820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.747531 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.748974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.748944 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:00.749105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.749007 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:02:00.749176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.749111 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:02:00.749176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.749146 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr" pod "51b09322-4337-4ff7-b0e4-bb3cc82436d5" (UID: "51b09322-4337-4ff7-b0e4-bb3cc82436d5"). InnerVolumeSpecName "kube-api-access-h4tlr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:00.848243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.848209 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.848243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.848246 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.848470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.848260 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.848470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.848271 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4tlr\" (UniqueName: \"kubernetes.io/projected/51b09322-4337-4ff7-b0e4-bb3cc82436d5-kube-api-access-h4tlr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:00.848470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:00.848286 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51b09322-4337-4ff7-b0e4-bb3cc82436d5-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:01.430139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.430028 5144 generic.go:356] "Generic (PLEG): container finished" podID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerID="48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c" exitCode=0 Nov 03 01:02:01.430139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.430102 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" Nov 03 01:02:01.430139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.430112 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" event={"ID":"51b09322-4337-4ff7-b0e4-bb3cc82436d5","Type":"ContainerDied","Data":"48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c"} Nov 03 01:02:01.430480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.430193 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p" event={"ID":"51b09322-4337-4ff7-b0e4-bb3cc82436d5","Type":"ContainerDied","Data":"810a9df64b1e9fff226d64fe007104f6fccf947a66cf1513318049a3af030089"} Nov 03 01:02:01.430480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.430219 5144 scope.go:122] "RemoveContainer" containerID="48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c" Nov 03 01:02:01.442933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.442912 5144 scope.go:122] "RemoveContainer" containerID="8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b" Nov 03 01:02:01.455540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.455511 5144 scope.go:122] "RemoveContainer" containerID="48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c" Nov 03 01:02:01.456019 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:01.455989 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c\": container with ID starting with 48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c not found: ID does not exist" containerID="48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c" Nov 03 01:02:01.456126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.456029 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c"} err="failed to get container status \"48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c\": rpc error: code = NotFound desc = could not find container \"48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c\": container with ID starting with 48088888f5d13506506ea0809d68d9c1398638bec9959c59f5728f2a2cda390c not found: ID does not exist" Nov 03 01:02:01.456126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.456048 5144 scope.go:122] "RemoveContainer" containerID="8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b" Nov 03 01:02:01.456386 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:01.456363 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b\": container with ID starting with 8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b not found: ID does not exist" containerID="8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b" Nov 03 01:02:01.456456 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.456391 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b"} err="failed to get container status \"8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b\": rpc error: code = NotFound desc = could not find container \"8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b\": container with ID starting with 8f0d4dfa3a028ca450fdb583791759a6bd314b9ce63c12e79d6379b75c84394b not found: ID does not exist" Nov 03 01:02:01.594967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.594929 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p"] Nov 03 01:02:01.645421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.645386 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-115ee5-openshift-default-5cf7448f97-pcx8p"] Nov 03 01:02:01.998542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:01.998505 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" path="/var/lib/kubelet/pods/51b09322-4337-4ff7-b0e4-bb3cc82436d5/volumes" Nov 03 01:02:02.981272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:02.981226 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g"] Nov 03 01:02:03.089911 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:03.089864 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2"] Nov 03 01:02:03.093891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:03.093858 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f7b976a3-mpcc2"] Nov 03 01:02:03.997940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:03.997902 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20315322-e2ce-46a6-96c7-f00ca1304cc3" path="/var/lib/kubelet/pods/20315322-e2ce-46a6-96c7-f00ca1304cc3/volumes" Nov 03 01:02:07.981979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:07.981926 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" podUID="44fe2b24-5688-4e2e-b910-b0f03c173953" containerName="payload-processing" containerID="cri-o://855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a" gracePeriod=25 Nov 03 01:02:08.282229 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.282195 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 01:02:08.442041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.441971 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume\") pod \"44fe2b24-5688-4e2e-b910-b0f03c173953\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " Nov 03 01:02:08.442255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.442154 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg\" (UniqueName: \"kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg\") pod \"44fe2b24-5688-4e2e-b910-b0f03c173953\" (UID: \"44fe2b24-5688-4e2e-b910-b0f03c173953\") " Nov 03 01:02:08.442536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.442505 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume" pod "44fe2b24-5688-4e2e-b910-b0f03c173953" (UID: "44fe2b24-5688-4e2e-b910-b0f03c173953"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:08.443123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.443098 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44fe2b24-5688-4e2e-b910-b0f03c173953-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:08.447590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.447555 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg" pod "44fe2b24-5688-4e2e-b910-b0f03c173953" (UID: "44fe2b24-5688-4e2e-b910-b0f03c173953"). InnerVolumeSpecName "kube-api-access-5f8vg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:08.500565 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.500477 5144 generic.go:356] "Generic (PLEG): container finished" podID="44fe2b24-5688-4e2e-b910-b0f03c173953" containerID="855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a" exitCode=0 Nov 03 01:02:08.500565 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.500547 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" Nov 03 01:02:08.500821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.500551 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" event={"ID":"44fe2b24-5688-4e2e-b910-b0f03c173953","Type":"ContainerDied","Data":"855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a"} Nov 03 01:02:08.500821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.500695 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g" event={"ID":"44fe2b24-5688-4e2e-b910-b0f03c173953","Type":"ContainerDied","Data":"9ab8ca763656597fab899e0c656ffe7ac25f51d9c836d21e40520c3f78724d66"} Nov 03 01:02:08.500821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.500720 5144 scope.go:122] "RemoveContainer" containerID="855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a" Nov 03 01:02:08.512560 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.512536 5144 scope.go:122] "RemoveContainer" containerID="855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a" Nov 03 01:02:08.512958 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:08.512937 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a\": container with ID starting with 855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a not found: ID does not exist" containerID="855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a" Nov 03 01:02:08.513026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.512967 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a"} err="failed to get container status \"855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a\": rpc error: code = NotFound desc = could not find container \"855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a\": container with ID starting with 855df39b34325e291b3c014d9324dfc6f37dae718dc805cc541883d3531fc04a not found: ID does not exist" Nov 03 01:02:08.544792 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.544754 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f8vg\" (UniqueName: \"kubernetes.io/projected/44fe2b24-5688-4e2e-b910-b0f03c173953-kube-api-access-5f8vg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:08.665768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.665732 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g"] Nov 03 01:02:08.669186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:08.669151 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-642e6a-c55d5b9f7-vxz9g"] Nov 03 01:02:09.997990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:09.997955 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44fe2b24-5688-4e2e-b910-b0f03c173953" path="/var/lib/kubelet/pods/44fe2b24-5688-4e2e-b910-b0f03c173953/volumes" Nov 03 01:02:12.157952 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:12.157905 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:02:13.047106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:13.047076 5144 scope.go:122] "RemoveContainer" containerID="f61ee819ecde046c6ff06565655357570708209daa1b35e8c721691685ad1adb" Nov 03 01:02:13.058586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:13.058560 5144 scope.go:122] "RemoveContainer" containerID="865ec5d2f572fad4e719e25f75be431a0c7a460cd672e74a893b6c78a3bae1cc" Nov 03 01:02:13.068220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:13.068196 5144 scope.go:122] "RemoveContainer" containerID="9ef698e64a19b3b9ebfb2eff0be150c71cb15741f189818f92b1b9c7fe008a8d" Nov 03 01:02:24.897559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:24.897524 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64"] Nov 03 01:02:29.898356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:29.898309 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" podUID="12aa25ab-ade3-4157-a88a-74b1ed30ee8d" containerName="payload-processing" containerID="cri-o://7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad" gracePeriod=25 Nov 03 01:02:30.215211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.215180 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 01:02:30.318887 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.318851 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume\") pod \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " Nov 03 01:02:30.319101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.318935 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57\" (UniqueName: \"kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57\") pod \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\" (UID: \"12aa25ab-ade3-4157-a88a-74b1ed30ee8d\") " Nov 03 01:02:30.319263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.319236 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume" pod "12aa25ab-ade3-4157-a88a-74b1ed30ee8d" (UID: "12aa25ab-ade3-4157-a88a-74b1ed30ee8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:30.321353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.321331 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57" pod "12aa25ab-ade3-4157-a88a-74b1ed30ee8d" (UID: "12aa25ab-ade3-4157-a88a-74b1ed30ee8d"). InnerVolumeSpecName "kube-api-access-6rq57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:30.420066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.420027 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:30.420066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.420059 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rq57\" (UniqueName: \"kubernetes.io/projected/12aa25ab-ade3-4157-a88a-74b1ed30ee8d-kube-api-access-6rq57\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:30.698447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.698410 5144 generic.go:356] "Generic (PLEG): container finished" podID="12aa25ab-ade3-4157-a88a-74b1ed30ee8d" containerID="7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad" exitCode=0 Nov 03 01:02:30.698684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.698459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" Nov 03 01:02:30.698684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.698495 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" event={"ID":"12aa25ab-ade3-4157-a88a-74b1ed30ee8d","Type":"ContainerDied","Data":"7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad"} Nov 03 01:02:30.698684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.698552 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64" event={"ID":"12aa25ab-ade3-4157-a88a-74b1ed30ee8d","Type":"ContainerDied","Data":"f4f4719c9085fbbb740af23726ab4edc969189e509419a78ad08a02977685c06"} Nov 03 01:02:30.698684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.698576 5144 scope.go:122] "RemoveContainer" containerID="7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad" Nov 03 01:02:30.709882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.709859 5144 scope.go:122] "RemoveContainer" containerID="7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad" Nov 03 01:02:30.710199 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:30.710177 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad\": container with ID starting with 7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad not found: ID does not exist" containerID="7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad" Nov 03 01:02:30.710250 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.710212 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad"} err="failed to get container status \"7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad\": rpc error: code = NotFound desc = could not find container \"7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad\": container with ID starting with 7161b6c2050eebda374e99e19c57a89a988f22b1d5a186729a925c34c4db68ad not found: ID does not exist" Nov 03 01:02:30.867147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.867114 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64"] Nov 03 01:02:30.917134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:30.917106 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-3b1d36-cbd7d5f4f-dqp64"] Nov 03 01:02:31.996969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:31.996931 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12aa25ab-ade3-4157-a88a-74b1ed30ee8d" path="/var/lib/kubelet/pods/12aa25ab-ade3-4157-a88a-74b1ed30ee8d/volumes" Nov 03 01:02:38.406076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406039 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8"] Nov 03 01:02:38.406808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406792 5144 cpu_manager.go:414] "containerMap: removing container" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" Nov 03 01:02:38.406868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406810 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" Nov 03 01:02:38.406868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406819 5144 cpu_manager.go:414] "containerMap: removing container" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" Nov 03 01:02:38.406868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406824 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" Nov 03 01:02:38.406868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406840 5144 cpu_manager.go:414] "containerMap: removing container" podUID="12aa25ab-ade3-4157-a88a-74b1ed30ee8d" containerName="payload-processing" Nov 03 01:02:38.406868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406845 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12aa25ab-ade3-4157-a88a-74b1ed30ee8d" containerName="payload-processing" Nov 03 01:02:38.407039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406883 5144 cpu_manager.go:414] "containerMap: removing container" podUID="44fe2b24-5688-4e2e-b910-b0f03c173953" containerName="payload-processing" Nov 03 01:02:38.407039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.406888 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44fe2b24-5688-4e2e-b910-b0f03c173953" containerName="payload-processing" Nov 03 01:02:38.407039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.407006 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="44fe2b24-5688-4e2e-b910-b0f03c173953" containerName="payload-processing" Nov 03 01:02:38.407039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.407017 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c6dc9be-ef7c-4aa7-aa6d-238dc7245803" containerName="istio-proxy" Nov 03 01:02:38.407039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.407029 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" Nov 03 01:02:38.407039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.407034 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="51b09322-4337-4ff7-b0e4-bb3cc82436d5" containerName="istio-proxy" Nov 03 01:02:38.407245 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.407042 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="12aa25ab-ade3-4157-a88a-74b1ed30ee8d" containerName="payload-processing" Nov 03 01:02:38.412560 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.412542 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.442929 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.442893 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.443060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.443008 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.443107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.443093 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.443192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.443180 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62f59\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.466149 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.466122 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8"] Nov 03 01:02:38.511488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.511456 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-e535ec6795f9\"" Nov 03 01:02:38.511685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.511503 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mxnb8\"" Nov 03 01:02:38.544052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.544015 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.544218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.544072 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62f59\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.544218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.544123 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.544218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.544203 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.544916 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.544884 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.545038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.544933 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.546783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.546766 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.600567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.600537 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62f59\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59\") pod \"payload-processing-external-model-5b8d9d8787-dsbc8\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:38.726434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:38.726327 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:39.110941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:39.110915 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8"] Nov 03 01:02:39.112450 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:02:39.112423 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b4699a_a6e6_4eec_9366_7e6a2987f785.slice/crio-815c34498832ff6ec41593311bcd716256789c806b5f7ebaed46947929611855 WatchSource:0}: Error finding container 815c34498832ff6ec41593311bcd716256789c806b5f7ebaed46947929611855: Status 404 returned error can't find the container with id 815c34498832ff6ec41593311bcd716256789c806b5f7ebaed46947929611855 Nov 03 01:02:39.781093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:39.781048 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" event={"ID":"97b4699a-a6e6-4eec-9366-7e6a2987f785","Type":"ContainerStarted","Data":"0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25"} Nov 03 01:02:39.781538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:39.781116 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" event={"ID":"97b4699a-a6e6-4eec-9366-7e6a2987f785","Type":"ContainerStarted","Data":"815c34498832ff6ec41593311bcd716256789c806b5f7ebaed46947929611855"} Nov 03 01:02:39.781538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:39.781144 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:39.894616 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:39.894563 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" podStartSLOduration=1.8945464300000001 podStartE2EDuration="1.89454643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:39.893407066 +0000 UTC m=+3450.039101547" watchObservedRunningTime="2026-11-03 01:02:39.89454643 +0000 UTC m=+3450.040240886" Nov 03 01:02:42.156352 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:42.156312 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:02:45.301945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.301912 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p"] Nov 03 01:02:45.306324 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.306305 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.358710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.358679 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 01:02:45.413584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.413421 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p"] Nov 03 01:02:45.433784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.433742 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.433962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.433859 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjzp\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.433962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.433907 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.433962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.433942 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.434078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.434011 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.434117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.434100 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.434153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.434127 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.434221 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.434209 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.434269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.434253 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.535984 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536066 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536118 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536157 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjzp\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536191 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536225 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536551 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536283 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536551 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536346 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536551 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536435 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536702 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536702 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536849 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536775 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.536903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.536880 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.537144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.537125 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.539426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.539397 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.539661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.539611 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.592553 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.592481 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.592832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.592804 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjzp\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:45.620953 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:45.620919 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:46.056622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.056590 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p"] Nov 03 01:02:46.096099 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:02:46.096071 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90f69fe_4d27_49c8_839b_8c363b471683.slice/crio-e87a869e826e9aeee93ae89c62cfe3c0355a75ae21f9620d5e97aa9989414ee9 WatchSource:0}: Error finding container e87a869e826e9aeee93ae89c62cfe3c0355a75ae21f9620d5e97aa9989414ee9: Status 404 returned error can't find the container with id e87a869e826e9aeee93ae89c62cfe3c0355a75ae21f9620d5e97aa9989414ee9 Nov 03 01:02:46.098736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.098697 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:02:46.098855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.098786 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:02:46.098855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.098832 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:02:46.839422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.839365 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" event={"ID":"e90f69fe-4d27-49c8-839b-8c363b471683","Type":"ContainerStarted","Data":"0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51"} Nov 03 01:02:46.840226 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.839439 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" event={"ID":"e90f69fe-4d27-49c8-839b-8c363b471683","Type":"ContainerStarted","Data":"e87a869e826e9aeee93ae89c62cfe3c0355a75ae21f9620d5e97aa9989414ee9"} Nov 03 01:02:46.963334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:46.963272 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podStartSLOduration=1.963253224 podStartE2EDuration="1.963253224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:46.959020432 +0000 UTC m=+3457.104714917" watchObservedRunningTime="2026-11-03 01:02:46.963253224 +0000 UTC m=+3457.108947682" Nov 03 01:02:47.626028 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:47.625992 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:47.645816 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:47.645774 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:47.646828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:47.646794 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:48.626657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:48.623044 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:48.626657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:48.623110 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:49.630657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:49.625033 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:49.630657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:49.625098 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:50.624033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:50.623995 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:50.624353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:50.624323 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:50.793802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:50.793771 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:02:50.965188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:50.965094 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-546ff98758-n7jm4"] Nov 03 01:02:51.626104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:51.626060 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:51.626418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:51.626385 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:52.621404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:52.621363 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:52.621980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:52.621430 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:52.982962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:52.982876 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j"] Nov 03 01:02:53.034395 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:53.034346 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c19567-vb55j"] Nov 03 01:02:53.622148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:53.622105 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" start-of-body= Nov 03 01:02:53.622738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:53.622188 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.41:15021/healthz/ready\": dial tcp 10.132.1.41:15021: connect: connection refused" Nov 03 01:02:53.997083 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:53.996989 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc78e05a-8fff-41c7-bbb5-b886c6167041" path="/var/lib/kubelet/pods/fc78e05a-8fff-41c7-bbb5-b886c6167041/volumes" Nov 03 01:02:54.959142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:54.959113 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:54.960490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:54.960461 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:54.961387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:54.961354 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:02:55.966796 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:55.966756 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" podUID="1379a12e-6e59-4529-b42c-c961ff42c78c" containerName="payload-processing" containerID="cri-o://764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971" gracePeriod=25 Nov 03 01:02:56.255319 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.255288 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:02:56.413574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.413542 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume\") pod \"1379a12e-6e59-4529-b42c-c961ff42c78c\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " Nov 03 01:02:56.413822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.413594 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume\") pod \"1379a12e-6e59-4529-b42c-c961ff42c78c\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " Nov 03 01:02:56.413822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.413659 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j\") pod \"1379a12e-6e59-4529-b42c-c961ff42c78c\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " Nov 03 01:02:56.413822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.413772 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials\") pod \"1379a12e-6e59-4529-b42c-c961ff42c78c\" (UID: \"1379a12e-6e59-4529-b42c-c961ff42c78c\") " Nov 03 01:02:56.414408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.414039 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume" pod "1379a12e-6e59-4529-b42c-c961ff42c78c" (UID: "1379a12e-6e59-4529-b42c-c961ff42c78c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:56.414408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.414373 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.414655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.414566 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume" pod "1379a12e-6e59-4529-b42c-c961ff42c78c" (UID: "1379a12e-6e59-4529-b42c-c961ff42c78c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:02:56.416216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.416188 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j" pod "1379a12e-6e59-4529-b42c-c961ff42c78c" (UID: "1379a12e-6e59-4529-b42c-c961ff42c78c"). InnerVolumeSpecName "kube-api-access-src7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:56.416351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.416228 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials" pod "1379a12e-6e59-4529-b42c-c961ff42c78c" (UID: "1379a12e-6e59-4529-b42c-c961ff42c78c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:02:56.515427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.515395 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1379a12e-6e59-4529-b42c-c961ff42c78c-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.515427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.515425 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-src7j\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-kube-api-access-src7j\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.515685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.515436 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1379a12e-6e59-4529-b42c-c961ff42c78c-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:02:56.946863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.946772 5144 generic.go:356] "Generic (PLEG): container finished" podID="1379a12e-6e59-4529-b42c-c961ff42c78c" containerID="764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971" exitCode=0 Nov 03 01:02:56.946863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.946825 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" Nov 03 01:02:56.946863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.946849 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" event={"ID":"1379a12e-6e59-4529-b42c-c961ff42c78c","Type":"ContainerDied","Data":"764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971"} Nov 03 01:02:56.947180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.946905 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-546ff98758-n7jm4" event={"ID":"1379a12e-6e59-4529-b42c-c961ff42c78c","Type":"ContainerDied","Data":"f6d845227f575487e8f4184d10579f33ec61026bd3a57f175ea677b27b701531"} Nov 03 01:02:56.947180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.946930 5144 scope.go:122] "RemoveContainer" containerID="764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971" Nov 03 01:02:56.960435 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.960410 5144 scope.go:122] "RemoveContainer" containerID="764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971" Nov 03 01:02:56.960823 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:02:56.960802 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971\": container with ID starting with 764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971 not found: ID does not exist" containerID="764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971" Nov 03 01:02:56.960917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:56.960833 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971"} err="failed to get container status \"764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971\": rpc error: code = NotFound desc = could not find container \"764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971\": container with ID starting with 764c9a3cdb4f98574c78fb08d5a4ef4e1d18af3644763ca46fd88a2e4570c971 not found: ID does not exist" Nov 03 01:02:57.110754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:57.110720 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-546ff98758-n7jm4"] Nov 03 01:02:57.114022 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:57.113998 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-546ff98758-n7jm4"] Nov 03 01:02:57.997722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:02:57.997685 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1379a12e-6e59-4529-b42c-c961ff42c78c" path="/var/lib/kubelet/pods/1379a12e-6e59-4529-b42c-c961ff42c78c/volumes" Nov 03 01:03:00.734677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.734609 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-68495976f-8hfk4"] Nov 03 01:03:00.735310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.735295 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1379a12e-6e59-4529-b42c-c961ff42c78c" containerName="payload-processing" Nov 03 01:03:00.735310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.735311 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1379a12e-6e59-4529-b42c-c961ff42c78c" containerName="payload-processing" Nov 03 01:03:00.735474 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.735464 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1379a12e-6e59-4529-b42c-c961ff42c78c" containerName="payload-processing" Nov 03 01:03:00.740481 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.740461 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.793541 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.793516 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68495976f-8hfk4"] Nov 03 01:03:00.838518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.838492 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-2302a5a10819\"" Nov 03 01:03:00.880823 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.880791 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.881005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.880930 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.881114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.881086 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbj68\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.881210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.881192 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.982528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.982488 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.982784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.982597 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbj68\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.982784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.982701 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.982784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.982778 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.983363 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.983306 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.983573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.983451 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:00.985596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:00.985537 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:01.040275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:01.040240 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbj68\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68\") pod \"payload-processing-external-model-68495976f-8hfk4\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:01.055100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:01.055073 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:01.445385 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:01.445358 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-68495976f-8hfk4"] Nov 03 01:03:01.447133 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:03:01.447100 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372f1149_53d4_4317_b340_b0c6ec57b2ff.slice/crio-b0771c37a5e30e9d52c3a614da9460cc46b28fe1947011fc98785f1ed42dd985 WatchSource:0}: Error finding container b0771c37a5e30e9d52c3a614da9460cc46b28fe1947011fc98785f1ed42dd985: Status 404 returned error can't find the container with id b0771c37a5e30e9d52c3a614da9460cc46b28fe1947011fc98785f1ed42dd985 Nov 03 01:03:02.002969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:02.002933 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" event={"ID":"372f1149-53d4-4317-b340-b0c6ec57b2ff","Type":"ContainerStarted","Data":"898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab"} Nov 03 01:03:02.003370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:02.002994 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" event={"ID":"372f1149-53d4-4317-b340-b0c6ec57b2ff","Type":"ContainerStarted","Data":"b0771c37a5e30e9d52c3a614da9460cc46b28fe1947011fc98785f1ed42dd985"} Nov 03 01:03:02.003370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:02.003026 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:02.117224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:02.117162 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" podStartSLOduration=2.117143364 podStartE2EDuration="2.117143364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:02.116182514 +0000 UTC m=+3472.261876991" watchObservedRunningTime="2026-11-03 01:03:02.117143364 +0000 UTC m=+3472.262837822" Nov 03 01:03:12.155776 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:12.155725 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:03:13.013611 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:13.013580 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:03:13.149923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:13.149882 5144 scope.go:122] "RemoveContainer" containerID="9ecb222291d118a599c6bf29f59f223140d6d4943b33d09b46f7c49f7e998032" Nov 03 01:03:13.165244 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:13.165220 5144 scope.go:122] "RemoveContainer" containerID="a245a5da5bb2908b18ac59e479bcdc5edfeb2ccf3d995424e4b7c18daab0f0f8" Nov 03 01:03:13.175093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:13.175074 5144 scope.go:122] "RemoveContainer" containerID="6a7f57d698e2d73b74eaa01389328052a513b4f7e73626651e4b9c3e6db4425d" Nov 03 01:03:13.190659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:13.190588 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8"] Nov 03 01:03:13.200465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:13.200438 5144 scope.go:122] "RemoveContainer" containerID="0992f39398c21011a0dfcc6ef5d5963669828084399b7142309bacb570faab68" Nov 03 01:03:18.198161 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.198094 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" podUID="97b4699a-a6e6-4eec-9366-7e6a2987f785" containerName="payload-processing" containerID="cri-o://0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25" gracePeriod=25 Nov 03 01:03:18.602678 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.602624 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:03:18.772013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.771943 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59\") pod \"97b4699a-a6e6-4eec-9366-7e6a2987f785\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " Nov 03 01:03:18.772317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.772293 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume\") pod \"97b4699a-a6e6-4eec-9366-7e6a2987f785\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " Nov 03 01:03:18.772476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.772460 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials\") pod \"97b4699a-a6e6-4eec-9366-7e6a2987f785\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " Nov 03 01:03:18.772731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.772710 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume\") pod \"97b4699a-a6e6-4eec-9366-7e6a2987f785\" (UID: \"97b4699a-a6e6-4eec-9366-7e6a2987f785\") " Nov 03 01:03:18.773437 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.773375 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume" pod "97b4699a-a6e6-4eec-9366-7e6a2987f785" (UID: "97b4699a-a6e6-4eec-9366-7e6a2987f785"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:18.773437 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.773397 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume" pod "97b4699a-a6e6-4eec-9366-7e6a2987f785" (UID: "97b4699a-a6e6-4eec-9366-7e6a2987f785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:18.774108 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.774086 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:18.774254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.774240 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97b4699a-a6e6-4eec-9366-7e6a2987f785-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:18.775676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.775649 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59" pod "97b4699a-a6e6-4eec-9366-7e6a2987f785" (UID: "97b4699a-a6e6-4eec-9366-7e6a2987f785"). InnerVolumeSpecName "kube-api-access-62f59". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:18.775883 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.775863 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials" pod "97b4699a-a6e6-4eec-9366-7e6a2987f785" (UID: "97b4699a-a6e6-4eec-9366-7e6a2987f785"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:18.876185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.876095 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62f59\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-kube-api-access-62f59\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:18.876185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:18.876126 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/97b4699a-a6e6-4eec-9366-7e6a2987f785-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:19.186923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.186831 5144 generic.go:356] "Generic (PLEG): container finished" podID="97b4699a-a6e6-4eec-9366-7e6a2987f785" containerID="0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25" exitCode=0 Nov 03 01:03:19.186923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.186872 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" Nov 03 01:03:19.186923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.186901 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" event={"ID":"97b4699a-a6e6-4eec-9366-7e6a2987f785","Type":"ContainerDied","Data":"0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25"} Nov 03 01:03:19.187179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.186958 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8" event={"ID":"97b4699a-a6e6-4eec-9366-7e6a2987f785","Type":"ContainerDied","Data":"815c34498832ff6ec41593311bcd716256789c806b5f7ebaed46947929611855"} Nov 03 01:03:19.187179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.186980 5144 scope.go:122] "RemoveContainer" containerID="0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25" Nov 03 01:03:19.199120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.198915 5144 scope.go:122] "RemoveContainer" containerID="0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25" Nov 03 01:03:19.199418 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:19.199277 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25\": container with ID starting with 0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25 not found: ID does not exist" containerID="0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25" Nov 03 01:03:19.199418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.199301 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25"} err="failed to get container status \"0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25\": rpc error: code = NotFound desc = could not find container \"0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25\": container with ID starting with 0087bf189339231a145f21787470bd9734be7a0321459d6a5e844aff2da5ea25 not found: ID does not exist" Nov 03 01:03:19.352463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.352421 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8"] Nov 03 01:03:19.402363 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.402320 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5b8d9d8787-dsbc8"] Nov 03 01:03:19.997717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:19.997683 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97b4699a-a6e6-4eec-9366-7e6a2987f785" path="/var/lib/kubelet/pods/97b4699a-a6e6-4eec-9366-7e6a2987f785/volumes" Nov 03 01:03:25.057135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.056168 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:03:25.057135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.056262 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:03:25.244479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.244394 5144 generic.go:356] "Generic (PLEG): container finished" podID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerID="bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3" exitCode=0 Nov 03 01:03:25.244479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.244460 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" event={"ID":"9006f61a-46ec-4dbb-94b0-afca485bd4e2","Type":"ContainerDied","Data":"bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3"} Nov 03 01:03:25.244948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.244932 5144 scope.go:122] "RemoveContainer" containerID="bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3" Nov 03 01:03:25.246707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.246680 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:25.246788 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.246753 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:25.246830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:25.246785 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:26.254779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:26.254734 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" event={"ID":"9006f61a-46ec-4dbb-94b0-afca485bd4e2","Type":"ContainerStarted","Data":"d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb"} Nov 03 01:03:26.839656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:26.839497 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:03:26.839907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:26.839741 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:03:26.839907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:26.839821 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:03:27.839532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:27.839485 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:03:27.840044 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:27.839564 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:03:28.839397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:28.839339 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:03:28.839688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:28.839417 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:03:29.840350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:29.840303 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:03:29.840919 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:29.840373 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:03:30.654070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.654035 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56"] Nov 03 01:03:30.654733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.654718 5144 cpu_manager.go:414] "containerMap: removing container" podUID="97b4699a-a6e6-4eec-9366-7e6a2987f785" containerName="payload-processing" Nov 03 01:03:30.654805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.654734 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97b4699a-a6e6-4eec-9366-7e6a2987f785" containerName="payload-processing" Nov 03 01:03:30.654873 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.654863 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="97b4699a-a6e6-4eec-9366-7e6a2987f785" containerName="payload-processing" Nov 03 01:03:30.664305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.664264 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.731957 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732008 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732052 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732100 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732160 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732185 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732259 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpdsk\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.734657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.732294 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.784032 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.783989 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56"] Nov 03 01:03:30.834439 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.834391 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.844827 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.844895 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845028 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpdsk\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845074 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845212 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845379 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845427 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845501 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.835360 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.845658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845575 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.846695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.845928 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.846695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.846095 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.846695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.846215 5144 patch_prober.go:29] interesting pod/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" start-of-body= Nov 03 01:03:30.846695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.846327 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.36:15021/healthz/ready\": dial tcp 10.132.1.36:15021: connect: connection refused" Nov 03 01:03:30.846695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.846435 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.848517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.848476 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.849864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.849837 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.902266 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.902239 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.902449 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.902269 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpdsk\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-9mv56\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:30.981418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:30.981329 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:31.447098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.447061 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56"] Nov 03 01:03:31.484208 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:03:31.484174 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod265ce86d_3dcc_456b_b248_614f50db0988.slice/crio-9e49b4622895aa4246b4f614f8b5555e449ef7f034255dc072f291e0a6c5679e WatchSource:0}: Error finding container 9e49b4622895aa4246b4f614f8b5555e449ef7f034255dc072f291e0a6c5679e: Status 404 returned error can't find the container with id 9e49b4622895aa4246b4f614f8b5555e449ef7f034255dc072f291e0a6c5679e Nov 03 01:03:31.487497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.487463 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:31.487677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.487557 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:31.487677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.487602 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:31.844666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.844601 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:03:31.845138 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.845120 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:03:31.846006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:31.845986 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:03:32.299291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.299244 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" event={"ID":"265ce86d-3dcc-456b-b248-614f50db0988","Type":"ContainerStarted","Data":"919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88"} Nov 03 01:03:32.299511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.299314 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" event={"ID":"265ce86d-3dcc-456b-b248-614f50db0988","Type":"ContainerStarted","Data":"9e49b4622895aa4246b4f614f8b5555e449ef7f034255dc072f291e0a6c5679e"} Nov 03 01:03:32.417248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.417188 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podStartSLOduration=2.4171708450000002 podStartE2EDuration="2.417170845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:32.414256604 +0000 UTC m=+3502.559951059" watchObservedRunningTime="2026-11-03 01:03:32.417170845 +0000 UTC m=+3502.562865302" Nov 03 01:03:32.665101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.664864 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw"] Nov 03 01:03:32.671123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.671096 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.708211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.708179 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw"] Nov 03 01:03:32.748050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.748010 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-64081d29aff2\"" Nov 03 01:03:32.775930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.775887 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hzhb\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.776131 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.776015 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.776188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.776162 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.776239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.776186 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.877942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.877902 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.877942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.877947 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.879237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.878052 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hzhb\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.879237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.878104 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.879237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.878764 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.879237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.878874 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.881120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.881094 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.946492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.946384 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hzhb\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb\") pod \"payload-processing-external-model-77b4974db5-fd5nw\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:32.982081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.982045 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:32.984039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.984011 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:32.984189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.984071 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:32.992412 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:32.992386 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:33.412758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:33.412663 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw"] Nov 03 01:03:33.423150 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:03:33.423113 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf971cc1f_71ca_4189_aaae_83f60f38375a.slice/crio-e3d04dbc85aa91786fa36010d8ac9074856a0caf4136a6e76cc8e18939783817 WatchSource:0}: Error finding container e3d04dbc85aa91786fa36010d8ac9074856a0caf4136a6e76cc8e18939783817: Status 404 returned error can't find the container with id e3d04dbc85aa91786fa36010d8ac9074856a0caf4136a6e76cc8e18939783817 Nov 03 01:03:33.982322 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:33.982284 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:33.982806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:33.982339 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:34.349659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:34.342259 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" event={"ID":"f971cc1f-71ca-4189-aaae-83f60f38375a","Type":"ContainerStarted","Data":"ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43"} Nov 03 01:03:34.349877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:34.349774 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:34.349877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:34.349813 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" event={"ID":"f971cc1f-71ca-4189-aaae-83f60f38375a","Type":"ContainerStarted","Data":"e3d04dbc85aa91786fa36010d8ac9074856a0caf4136a6e76cc8e18939783817"} Nov 03 01:03:34.456018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:34.455958 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" podStartSLOduration=2.455944143 podStartE2EDuration="2.455944143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:34.454591533 +0000 UTC m=+3504.600286004" watchObservedRunningTime="2026-11-03 01:03:34.455944143 +0000 UTC m=+3504.601638612" Nov 03 01:03:34.982248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:34.982206 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:34.982466 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:34.982268 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:35.414702 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:35.411934 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64081d29aff2: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:35.414702 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:35.411982 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:35.414702 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:35.412053 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials podName:f971cc1f-71ca-4189-aaae-83f60f38375a nodeName:}" failed. No retries permitted until 2026-11-03 01:03:35.912018126 +0000 UTC m=+3506.057712578 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials") pod "payload-processing-external-model-77b4974db5-fd5nw" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a") : secret "agc-cred-64081d29aff2" not found Nov 03 01:03:35.930325 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:35.929848 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64081d29aff2: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:35.930325 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:35.929893 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:35.930325 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:35.929953 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials podName:f971cc1f-71ca-4189-aaae-83f60f38375a nodeName:}" failed. No retries permitted until 2026-11-03 01:03:36.929934316 +0000 UTC m=+3507.075628756 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials") pod "payload-processing-external-model-77b4974db5-fd5nw" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a") : secret "agc-cred-64081d29aff2" not found Nov 03 01:03:35.982755 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:35.982701 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:35.983180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:35.982777 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:36.959068 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:36.959018 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64081d29aff2: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:36.959068 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:36.959073 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:36.959319 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:36.959153 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials podName:f971cc1f-71ca-4189-aaae-83f60f38375a nodeName:}" failed. No retries permitted until 2026-11-03 01:03:38.959132283 +0000 UTC m=+3509.104826730 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials") pod "payload-processing-external-model-77b4974db5-fd5nw" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a") : secret "agc-cred-64081d29aff2" not found Nov 03 01:03:36.981854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:36.981813 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:36.982084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:36.981872 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:37.982712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:37.982667 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:37.983141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:37.982745 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:38.075614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:38.075577 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw"] Nov 03 01:03:38.079782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:38.079754 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:38.981786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:38.981738 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" start-of-body= Nov 03 01:03:38.981990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:38.981802 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.43:15021/healthz/ready\": dial tcp 10.132.1.43:15021: connect: connection refused" Nov 03 01:03:38.990905 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:38.990852 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64081d29aff2: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:38.990905 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:38.990905 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:38.991304 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:38.990963 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials podName:f971cc1f-71ca-4189-aaae-83f60f38375a nodeName:}" failed. No retries permitted until 2026-11-03 01:03:42.990947625 +0000 UTC m=+3513.136642059 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials") pod "payload-processing-external-model-77b4974db5-fd5nw" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a") : secret "agc-cred-64081d29aff2" not found Nov 03 01:03:40.551212 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:40.551176 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:40.552675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:40.551970 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:40.553162 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:40.553144 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:03:42.160857 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:42.160820 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:03:43.066048 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:43.066013 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-64081d29aff2: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:43.066048 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:43.066050 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw: secret "agc-cred-64081d29aff2" not found Nov 03 01:03:43.066369 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:43.066119 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials podName:f971cc1f-71ca-4189-aaae-83f60f38375a nodeName:}" failed. No retries permitted until 2026-11-03 01:03:51.066100577 +0000 UTC m=+3521.211795011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials") pod "payload-processing-external-model-77b4974db5-fd5nw" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a") : secret "agc-cred-64081d29aff2" not found Nov 03 01:03:43.076297 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.076253 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" podUID="f971cc1f-71ca-4189-aaae-83f60f38375a" containerName="payload-processing" containerID="cri-o://ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43" gracePeriod=25 Nov 03 01:03:43.363564 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.363528 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:43.468357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.468321 5144 generic.go:356] "Generic (PLEG): container finished" podID="f971cc1f-71ca-4189-aaae-83f60f38375a" containerID="ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43" exitCode=0 Nov 03 01:03:43.468570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.468370 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" Nov 03 01:03:43.468570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.468405 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" event={"ID":"f971cc1f-71ca-4189-aaae-83f60f38375a","Type":"ContainerDied","Data":"ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43"} Nov 03 01:03:43.468570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.468462 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw" event={"ID":"f971cc1f-71ca-4189-aaae-83f60f38375a","Type":"ContainerDied","Data":"e3d04dbc85aa91786fa36010d8ac9074856a0caf4136a6e76cc8e18939783817"} Nov 03 01:03:43.468570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.468490 5144 scope.go:122] "RemoveContainer" containerID="ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43" Nov 03 01:03:43.472100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.472076 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume\") pod \"f971cc1f-71ca-4189-aaae-83f60f38375a\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " Nov 03 01:03:43.472203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.472148 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume\") pod \"f971cc1f-71ca-4189-aaae-83f60f38375a\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " Nov 03 01:03:43.472288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.472271 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb\") pod \"f971cc1f-71ca-4189-aaae-83f60f38375a\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " Nov 03 01:03:43.472434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.472414 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials\") pod \"f971cc1f-71ca-4189-aaae-83f60f38375a\" (UID: \"f971cc1f-71ca-4189-aaae-83f60f38375a\") " Nov 03 01:03:43.472488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.472450 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume" pod "f971cc1f-71ca-4189-aaae-83f60f38375a" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:43.472549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.472518 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume" pod "f971cc1f-71ca-4189-aaae-83f60f38375a" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:03:43.473087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.473060 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:43.473235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.473090 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f971cc1f-71ca-4189-aaae-83f60f38375a-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:43.474700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.474676 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb" pod "f971cc1f-71ca-4189-aaae-83f60f38375a" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a"). InnerVolumeSpecName "kube-api-access-6hzhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:43.475065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.475040 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials" pod "f971cc1f-71ca-4189-aaae-83f60f38375a" (UID: "f971cc1f-71ca-4189-aaae-83f60f38375a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:03:43.481945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.481918 5144 scope.go:122] "RemoveContainer" containerID="ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43" Nov 03 01:03:43.482314 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:03:43.482288 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43\": container with ID starting with ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43 not found: ID does not exist" containerID="ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43" Nov 03 01:03:43.482455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.482324 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43"} err="failed to get container status \"ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43\": rpc error: code = NotFound desc = could not find container \"ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43\": container with ID starting with ccea18622a642ec4df5e39e8df5cae6d4b84b40a6cc1e2b1075559f0dae9de43 not found: ID does not exist" Nov 03 01:03:43.575154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.575104 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hzhb\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-kube-api-access-6hzhb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:43.575154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.575150 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f971cc1f-71ca-4189-aaae-83f60f38375a-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:03:43.892718 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.892681 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw"] Nov 03 01:03:43.940387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.940348 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-77b4974db5-fd5nw"] Nov 03 01:03:43.998471 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:43.998433 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f971cc1f-71ca-4189-aaae-83f60f38375a" path="/var/lib/kubelet/pods/f971cc1f-71ca-4189-aaae-83f60f38375a/volumes" Nov 03 01:03:45.795368 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.795325 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8"] Nov 03 01:03:45.796072 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.796058 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f971cc1f-71ca-4189-aaae-83f60f38375a" containerName="payload-processing" Nov 03 01:03:45.796127 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.796074 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f971cc1f-71ca-4189-aaae-83f60f38375a" containerName="payload-processing" Nov 03 01:03:45.796220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.796211 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f971cc1f-71ca-4189-aaae-83f60f38375a" containerName="payload-processing" Nov 03 01:03:45.801336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.801306 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.908851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.908807 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.908851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.908869 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.908993 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.909083 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.909152 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.909184 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjlx8\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.909370 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.909429 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:45.909498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:45.909460 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.006809 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.006769 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8"] Nov 03 01:03:46.010913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.010877 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.011183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.011165 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.011318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.011303 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.011474 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.011461 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.011619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.011605 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.011759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.011744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjlx8\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.012002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.011987 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.012137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.012123 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.012239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.012227 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.012797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.012773 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.016223 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.016189 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.017239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.017209 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.017486 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.017465 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.018073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.018050 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.018808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.018786 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.020449 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.020423 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.073066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.072991 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.074259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.074240 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjlx8\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-928s8\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.117804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.117768 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:46.560959 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.560920 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8"] Nov 03 01:03:46.563160 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:03:46.563127 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1711b6b_c6fc_4afc_aa11_1c14f46017c1.slice/crio-40707da4a1d6193264c8004bc50aa38f03a56152d31e7cda4db93995a9828c32 WatchSource:0}: Error finding container 40707da4a1d6193264c8004bc50aa38f03a56152d31e7cda4db93995a9828c32: Status 404 returned error can't find the container with id 40707da4a1d6193264c8004bc50aa38f03a56152d31e7cda4db93995a9828c32 Nov 03 01:03:46.566112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.566075 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:46.566214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.566151 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:46.566214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:46.566180 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:03:47.273981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.273939 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76"] Nov 03 01:03:47.278967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.278940 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.340892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.340648 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-4db6f5-dockercfg-j2zft\"" Nov 03 01:03:47.340892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.340741 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-4db6f5\"" Nov 03 01:03:47.351656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.345767 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76"] Nov 03 01:03:47.504660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.503142 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh"] Nov 03 01:03:47.553722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.517880 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" event={"ID":"f1711b6b-c6fc-4afc-aa11-1c14f46017c1","Type":"ContainerStarted","Data":"6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce"} Nov 03 01:03:47.553722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.517934 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh"] Nov 03 01:03:47.553722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.517968 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" event={"ID":"f1711b6b-c6fc-4afc-aa11-1c14f46017c1","Type":"ContainerStarted","Data":"40707da4a1d6193264c8004bc50aa38f03a56152d31e7cda4db93995a9828c32"} Nov 03 01:03:47.553722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.518065 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.559542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.559509 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkbjx\" (UniqueName: \"kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx\") pod \"payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.559771 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.559619 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvjn\" (UniqueName: \"kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn\") pod \"payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.559771 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.559684 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.559771 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.559711 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.661768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.661724 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkbjx\" (UniqueName: \"kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx\") pod \"payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.662038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.661827 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvjn\" (UniqueName: \"kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn\") pod \"payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.662038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.661856 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.662038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.661872 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.662720 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.662700 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.662846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.662824 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.746527 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.746441 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvjn\" (UniqueName: \"kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn\") pod \"payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:47.758401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.758344 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkbjx\" (UniqueName: \"kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx\") pod \"payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.765592 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.765522 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podStartSLOduration=2.765500752 podStartE2EDuration="2.765500752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:47.650153366 +0000 UTC m=+3517.795847822" watchObservedRunningTime="2026-11-03 01:03:47.765500752 +0000 UTC m=+3517.911195211" Nov 03 01:03:47.871278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.871182 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:47.895387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:47.895349 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:48.137724 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.123732 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:48.149899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.149842 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:48.150253 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.150206 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:48.309689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.309656 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh"] Nov 03 01:03:48.311032 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:03:48.311005 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0350ec1b_16ff_47ec_acbe_f666ecc05ef9.slice/crio-88e179f980d2301c0ec3ef14af63042b271c3792dbf26d5d1fdda190c48aaca1 WatchSource:0}: Error finding container 88e179f980d2301c0ec3ef14af63042b271c3792dbf26d5d1fdda190c48aaca1: Status 404 returned error can't find the container with id 88e179f980d2301c0ec3ef14af63042b271c3792dbf26d5d1fdda190c48aaca1 Nov 03 01:03:48.363937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.363904 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76"] Nov 03 01:03:48.554807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.554754 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" event={"ID":"e8395528-f715-4eb3-acc4-b98ceaa553a0","Type":"ContainerStarted","Data":"038541b2082c221038183fde6b2691931efbd0d5635d39b945a94b7944b86813"} Nov 03 01:03:48.557218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.557183 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" event={"ID":"0350ec1b-16ff-47ec-acbe-f666ecc05ef9","Type":"ContainerStarted","Data":"0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c"} Nov 03 01:03:48.557383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.557254 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" event={"ID":"0350ec1b-16ff-47ec-acbe-f666ecc05ef9","Type":"ContainerStarted","Data":"88e179f980d2301c0ec3ef14af63042b271c3792dbf26d5d1fdda190c48aaca1"} Nov 03 01:03:48.670996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:48.670933 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" podStartSLOduration=1.670914437 podStartE2EDuration="1.670914437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:48.670892477 +0000 UTC m=+3518.816586933" watchObservedRunningTime="2026-11-03 01:03:48.670914437 +0000 UTC m=+3518.816608895" Nov 03 01:03:49.123585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:49.123122 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:49.123585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:49.123189 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:49.567999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:49.567961 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" event={"ID":"e8395528-f715-4eb3-acc4-b98ceaa553a0","Type":"ContainerStarted","Data":"3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9"} Nov 03 01:03:49.568736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:49.568717 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:03:49.686935 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:49.686876 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" podStartSLOduration=2.686861904 podStartE2EDuration="2.686861904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:03:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:03:49.68239204 +0000 UTC m=+3519.828086519" watchObservedRunningTime="2026-11-03 01:03:49.686861904 +0000 UTC m=+3519.832556441" Nov 03 01:03:50.127659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:50.122950 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:50.127659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:50.123016 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:50.576170 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:50.576136 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:03:51.131659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:51.127027 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:51.131659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:51.127094 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:52.122657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:52.120120 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:52.122657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:52.120194 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:53.118140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:53.118104 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:53.118372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:53.118161 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:54.119280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:54.119220 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" start-of-body= Nov 03 01:03:54.119839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:54.119301 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.45:15021/healthz/ready\": dial tcp 10.132.1.45:15021: connect: connection refused" Nov 03 01:03:55.429370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:55.429332 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:55.430373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:55.430344 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:03:55.431371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:03:55.431349 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:04:00.580303 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:00.580266 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:04:00.907138 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:00.907047 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp"] Nov 03 01:04:00.911829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:00.911807 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:00.996361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:00.996326 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp"] Nov 03 01:04:01.064714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.064623 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.064948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.064787 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.064948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.064832 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.064948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.064892 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.065140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.064960 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.065140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.065032 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.065140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.065106 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlcp4\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.065267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.065148 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.065267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.065175 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166504 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166561 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166584 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166616 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166734 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166801 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166851 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlcp4\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166882 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.166942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.166912 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.167284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.167073 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.167284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.167245 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.167460 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.167438 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.167539 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.167462 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.167793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.167772 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.169480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.169456 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.169600 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.169576 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.223537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.223505 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.223764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.223576 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlcp4\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.227490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.227456 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:01.587434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.587400 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:04:01.668114 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:04:01.668082 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc02995a_bc90_4b09_8b2a_2d9e69875588.slice/crio-9ab5ccc20a9d4625aeaaa95e446034e27ccf7ea4caadde2673c3309acc87f7bd WatchSource:0}: Error finding container 9ab5ccc20a9d4625aeaaa95e446034e27ccf7ea4caadde2673c3309acc87f7bd: Status 404 returned error can't find the container with id 9ab5ccc20a9d4625aeaaa95e446034e27ccf7ea4caadde2673c3309acc87f7bd Nov 03 01:04:01.668410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.668386 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp"] Nov 03 01:04:01.670947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.670913 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:01.671082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.670991 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:01.671082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:01.671035 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:02.683451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:02.683405 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" event={"ID":"cc02995a-bc90-4b09-8b2a-2d9e69875588","Type":"ContainerStarted","Data":"a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac"} Nov 03 01:04:02.684113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:02.683515 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" event={"ID":"cc02995a-bc90-4b09-8b2a-2d9e69875588","Type":"ContainerStarted","Data":"9ab5ccc20a9d4625aeaaa95e446034e27ccf7ea4caadde2673c3309acc87f7bd"} Nov 03 01:04:02.806373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:02.806308 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podStartSLOduration=2.806291025 podStartE2EDuration="2.806291025s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:02.803811667 +0000 UTC m=+3532.949506137" watchObservedRunningTime="2026-11-03 01:04:02.806291025 +0000 UTC m=+3532.951985480" Nov 03 01:04:03.230105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:03.230066 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:03.244825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:03.244778 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:03.245137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:03.245099 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:04.229658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:04.229130 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:04.229658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:04.229230 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:05.228172 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:05.228126 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:05.228525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:05.228490 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:06.229182 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:06.229137 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:06.229882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:06.229846 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:07.230212 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:07.230172 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:07.231318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:07.230886 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:08.228985 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:08.228937 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:08.229251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:08.229023 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:09.228479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:09.228437 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" start-of-body= Nov 03 01:04:09.228976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:09.228522 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.48:15021/healthz/ready\": dial tcp 10.132.1.48:15021: connect: connection refused" Nov 03 01:04:10.414403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:10.414363 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:10.415043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:10.414826 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:10.415993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:10.415968 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:04:12.234903 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:04:12.234861 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:04:22.818136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.818029 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w"] Nov 03 01:04:22.823214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.823183 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.877195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.877144 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk8kn\" (UniqueName: \"kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.877422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.877293 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.877422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.877348 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.881522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.881490 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w"] Nov 03 01:04:22.924819 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.924779 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:04:22.924819 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.924791 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:04:22.978605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.978561 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk8kn\" (UniqueName: \"kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.978834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.978702 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.978834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.978756 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.979489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.979461 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:22.981855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:22.981826 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:23.036197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.036155 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk8kn\" (UniqueName: \"kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn\") pod \"maas-api-revoke-sub-d1353522-cjq9w\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:23.140886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.140782 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:23.568005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.567966 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w"] Nov 03 01:04:23.568507 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:04:23.568479 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03237d99_29ab_4d00_b0a5_065ce884bc6e.slice/crio-cdd6dbba8a785e769ccdb3ed23826dc594ac1043417f0e6e503b7070e1509bd1 WatchSource:0}: Error finding container cdd6dbba8a785e769ccdb3ed23826dc594ac1043417f0e6e503b7070e1509bd1: Status 404 returned error can't find the container with id cdd6dbba8a785e769ccdb3ed23826dc594ac1043417f0e6e503b7070e1509bd1 Nov 03 01:04:23.944087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.944045 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895"] Nov 03 01:04:23.948786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.948758 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:23.990585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.990546 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:23.990827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.990613 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:23.990901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:23.990838 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlrsg\" (UniqueName: \"kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.004545 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.004504 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895"] Nov 03 01:04:24.081790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.081689 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" event={"ID":"03237d99-29ab-4d00-b0a5-065ce884bc6e","Type":"ContainerStarted","Data":"5027a9f089b2306e4ae1bdb2ceab17aec7cbcd175c89d48099df2f9393531245"} Nov 03 01:04:24.081790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.081762 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" event={"ID":"03237d99-29ab-4d00-b0a5-065ce884bc6e","Type":"ContainerStarted","Data":"cdd6dbba8a785e769ccdb3ed23826dc594ac1043417f0e6e503b7070e1509bd1"} Nov 03 01:04:24.092010 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.091975 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.092181 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.092143 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlrsg\" (UniqueName: \"kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.092299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.092281 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.092947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.092924 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.094842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.094817 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.150087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.150052 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlrsg\" (UniqueName: \"kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg\") pod \"maas-api-revoke-sub-5694f383-gz895\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.197018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.196960 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" podStartSLOduration=2.196946649 podStartE2EDuration="2.196946649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:24.194872503 +0000 UTC m=+3554.340566959" watchObservedRunningTime="2026-11-03 01:04:24.196946649 +0000 UTC m=+3554.342641104" Nov 03 01:04:24.263750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.263710 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:24.654328 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:24.654298 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895"] Nov 03 01:04:24.655368 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:04:24.655341 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5bcbff5_8926_47ae_b7e9_7a8cccec0cf3.slice/crio-54a821f5f3ddaadd736205d1b6baa0eb31ece186e876ec2a835505a7c7d6d00d WatchSource:0}: Error finding container 54a821f5f3ddaadd736205d1b6baa0eb31ece186e876ec2a835505a7c7d6d00d: Status 404 returned error can't find the container with id 54a821f5f3ddaadd736205d1b6baa0eb31ece186e876ec2a835505a7c7d6d00d Nov 03 01:04:25.092782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:25.092744 5144 generic.go:356] "Generic (PLEG): container finished" podID="03237d99-29ab-4d00-b0a5-065ce884bc6e" containerID="5027a9f089b2306e4ae1bdb2ceab17aec7cbcd175c89d48099df2f9393531245" exitCode=0 Nov 03 01:04:25.093277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:25.092843 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" event={"ID":"03237d99-29ab-4d00-b0a5-065ce884bc6e","Type":"ContainerDied","Data":"5027a9f089b2306e4ae1bdb2ceab17aec7cbcd175c89d48099df2f9393531245"} Nov 03 01:04:25.094733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:25.094697 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" event={"ID":"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3","Type":"ContainerStarted","Data":"5c2fb708dd2582db612c7c3f9f5e4d2d8fd0a2bb1cac0f7c07ba16b55721bcaf"} Nov 03 01:04:25.094898 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:25.094755 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" event={"ID":"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3","Type":"ContainerStarted","Data":"54a821f5f3ddaadd736205d1b6baa0eb31ece186e876ec2a835505a7c7d6d00d"} Nov 03 01:04:25.339033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:25.338975 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" podStartSLOduration=2.338956223 podStartE2EDuration="2.338956223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:25.337153594 +0000 UTC m=+3555.482848068" watchObservedRunningTime="2026-11-03 01:04:25.338956223 +0000 UTC m=+3555.484650679" Nov 03 01:04:26.104971 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.104928 5144 generic.go:356] "Generic (PLEG): container finished" podID="c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" containerID="5c2fb708dd2582db612c7c3f9f5e4d2d8fd0a2bb1cac0f7c07ba16b55721bcaf" exitCode=0 Nov 03 01:04:26.105516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.105011 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" event={"ID":"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3","Type":"ContainerDied","Data":"5c2fb708dd2582db612c7c3f9f5e4d2d8fd0a2bb1cac0f7c07ba16b55721bcaf"} Nov 03 01:04:26.298620 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.298588 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:26.434960 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.434857 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn\" (UniqueName: \"kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn\") pod \"03237d99-29ab-4d00-b0a5-065ce884bc6e\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " Nov 03 01:04:26.435141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.435029 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca\") pod \"03237d99-29ab-4d00-b0a5-065ce884bc6e\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " Nov 03 01:04:26.435141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.435066 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca\") pod \"03237d99-29ab-4d00-b0a5-065ce884bc6e\" (UID: \"03237d99-29ab-4d00-b0a5-065ce884bc6e\") " Nov 03 01:04:26.435470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.435440 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca" pod "03237d99-29ab-4d00-b0a5-065ce884bc6e" (UID: "03237d99-29ab-4d00-b0a5-065ce884bc6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:26.435952 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.435878 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:26.437599 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.437574 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn" pod "03237d99-29ab-4d00-b0a5-065ce884bc6e" (UID: "03237d99-29ab-4d00-b0a5-065ce884bc6e"). InnerVolumeSpecName "kube-api-access-dk8kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:26.438018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.437991 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca" pod "03237d99-29ab-4d00-b0a5-065ce884bc6e" (UID: "03237d99-29ab-4d00-b0a5-065ce884bc6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:26.536901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.536854 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dk8kn\" (UniqueName: \"kubernetes.io/projected/03237d99-29ab-4d00-b0a5-065ce884bc6e-kube-api-access-dk8kn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:26.536901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:26.536891 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03237d99-29ab-4d00-b0a5-065ce884bc6e-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:27.118562 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.118528 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" Nov 03 01:04:27.119043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.118567 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w" event={"ID":"03237d99-29ab-4d00-b0a5-065ce884bc6e","Type":"ContainerDied","Data":"cdd6dbba8a785e769ccdb3ed23826dc594ac1043417f0e6e503b7070e1509bd1"} Nov 03 01:04:27.119043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.118674 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd6dbba8a785e769ccdb3ed23826dc594ac1043417f0e6e503b7070e1509bd1" Nov 03 01:04:27.294061 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.294023 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:27.349566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.349527 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca\") pod \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " Nov 03 01:04:27.349813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.349596 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca\") pod \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " Nov 03 01:04:27.349813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.349790 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg\" (UniqueName: \"kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg\") pod \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\" (UID: \"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3\") " Nov 03 01:04:27.350060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.350038 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca" pod "c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" (UID: "c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:04:27.350404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.350385 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:27.352130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.352100 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca" pod "c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" (UID: "c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:04:27.352130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.352116 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg" pod "c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" (UID: "c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3"). InnerVolumeSpecName "kube-api-access-dlrsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:04:27.462750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.453721 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:27.462750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:27.461967 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlrsg\" (UniqueName: \"kubernetes.io/projected/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3-kube-api-access-dlrsg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:04:28.129377 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:28.129339 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" Nov 03 01:04:28.129844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:28.129332 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895" event={"ID":"c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3","Type":"ContainerDied","Data":"54a821f5f3ddaadd736205d1b6baa0eb31ece186e876ec2a835505a7c7d6d00d"} Nov 03 01:04:28.129844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:28.129438 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54a821f5f3ddaadd736205d1b6baa0eb31ece186e876ec2a835505a7c7d6d00d" Nov 03 01:04:40.959866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.959823 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk"] Nov 03 01:04:40.960561 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.960546 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" containerName="revoke-keys" Nov 03 01:04:40.960607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.960563 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" containerName="revoke-keys" Nov 03 01:04:40.960607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.960596 5144 cpu_manager.go:414] "containerMap: removing container" podUID="03237d99-29ab-4d00-b0a5-065ce884bc6e" containerName="revoke-keys" Nov 03 01:04:40.960607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.960601 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03237d99-29ab-4d00-b0a5-065ce884bc6e" containerName="revoke-keys" Nov 03 01:04:40.960757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.960749 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="03237d99-29ab-4d00-b0a5-065ce884bc6e" containerName="revoke-keys" Nov 03 01:04:40.960799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.960765 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" containerName="revoke-keys" Nov 03 01:04:40.965716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:40.965694 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.017538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.017497 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.017917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.017891 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.018110 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.018092 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.018298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.018278 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.018502 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.018484 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.018682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.018662 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zbq\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.018924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.018904 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.019138 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.019119 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.019412 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.019395 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.030435 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.030391 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-244924-openshift-default-dockercfg-h6zdl\"" Nov 03 01:04:41.082980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.082940 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk"] Nov 03 01:04:41.123014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.122955 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.123485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.123390 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zbq\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.123720 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.123698 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.123907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.123891 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.124090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.124073 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.124260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.124244 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.124414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.124397 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.124576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.124545 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.125081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.125039 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.126377 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.125076 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.126609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.124462 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.126810 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.125832 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.126810 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.126063 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.127014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.126988 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.129930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.129904 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.144025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.143985 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.180811 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.180774 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.183118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.183092 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zbq\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq\") pod \"e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.281079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.280795 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:41.326051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.326012 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn"] Nov 03 01:04:41.332190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.332158 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.389101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.389051 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn"] Nov 03 01:04:41.431914 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.431815 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wptpl\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.431914 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.431884 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.432210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.432017 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.432465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.432438 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.534733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.534600 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.534733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.534694 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.534966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.534805 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.534966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.534958 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wptpl\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.535785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.535751 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.535904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.535798 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.537673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.537613 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.591560 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.591519 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wptpl\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl\") pod \"payload-processing-external-model-6d78f968dc-mhwhn\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.661403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.661359 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:41.732176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.731590 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk"] Nov 03 01:04:41.735089 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:04:41.735055 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod098684db_b723_478b_a1a4_3907bd821719.slice/crio-7016388292e3465edf8b30f5b5cfee42aa265a45e169325655b1f4fd29b94732 WatchSource:0}: Error finding container 7016388292e3465edf8b30f5b5cfee42aa265a45e169325655b1f4fd29b94732: Status 404 returned error can't find the container with id 7016388292e3465edf8b30f5b5cfee42aa265a45e169325655b1f4fd29b94732 Nov 03 01:04:41.739295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.739244 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:41.739438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.739367 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:41.739438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:41.739419 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:04:42.070925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.070893 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn"] Nov 03 01:04:42.072414 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:04:42.072387 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f17efb3_775c_496c_9a35_4a1f1abaf9e9.slice/crio-79b8c0ba11b1a68ad143ed0641e83bfd94860abef77a3369ce9d21ad2f62fa2e WatchSource:0}: Error finding container 79b8c0ba11b1a68ad143ed0641e83bfd94860abef77a3369ce9d21ad2f62fa2e: Status 404 returned error can't find the container with id 79b8c0ba11b1a68ad143ed0641e83bfd94860abef77a3369ce9d21ad2f62fa2e Nov 03 01:04:42.169673 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:04:42.169592 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:04:42.300588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.300543 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" event={"ID":"098684db-b723-478b-a1a4-3907bd821719","Type":"ContainerStarted","Data":"26e375e1ef3df8e8bd2feb3fa19076041e46abc643d900a82dc8c8f2bba47af8"} Nov 03 01:04:42.300843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.300616 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" event={"ID":"098684db-b723-478b-a1a4-3907bd821719","Type":"ContainerStarted","Data":"7016388292e3465edf8b30f5b5cfee42aa265a45e169325655b1f4fd29b94732"} Nov 03 01:04:42.302650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.302603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" event={"ID":"2f17efb3-775c-496c-9a35-4a1f1abaf9e9","Type":"ContainerStarted","Data":"863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a"} Nov 03 01:04:42.302817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.302664 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" event={"ID":"2f17efb3-775c-496c-9a35-4a1f1abaf9e9","Type":"ContainerStarted","Data":"79b8c0ba11b1a68ad143ed0641e83bfd94860abef77a3369ce9d21ad2f62fa2e"} Nov 03 01:04:42.302817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.302714 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:04:42.419507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.419382 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podStartSLOduration=2.419364394 podStartE2EDuration="2.419364394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:42.415870403 +0000 UTC m=+3572.561564879" watchObservedRunningTime="2026-11-03 01:04:42.419364394 +0000 UTC m=+3572.565058848" Nov 03 01:04:42.545415 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:42.545342 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podStartSLOduration=1.545317663 podStartE2EDuration="1.545317663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:42.536049904 +0000 UTC m=+3572.681744389" watchObservedRunningTime="2026-11-03 01:04:42.545317663 +0000 UTC m=+3572.691012117" Nov 03 01:04:43.281955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:43.281917 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:43.289000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:43.288970 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:43.311383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:43.311357 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:43.312260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:43.312230 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:04:45.816765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.816728 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4"] Nov 03 01:04:45.822859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.822830 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:45.882062 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.882028 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4"] Nov 03 01:04:45.951666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.948940 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:45.951666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.949005 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkj47\" (UniqueName: \"kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47\") pod \"payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:45.963655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.956098 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-67b917-dockercfg-x4ggq\"" Nov 03 01:04:45.982658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:45.973924 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-67b917\"" Nov 03 01:04:46.051295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:46.051244 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:46.051500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:46.051308 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkj47\" (UniqueName: \"kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47\") pod \"payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:46.053746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:46.053711 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:46.109379 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:46.109298 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkj47\" (UniqueName: \"kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47\") pod \"payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:46.139438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:46.139390 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:46.531133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:46.531099 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4"] Nov 03 01:04:46.531265 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:04:46.531106 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136d4313_d15e_4903_b21a_c0d4bfed3610.slice/crio-fd62ec883d1e08954ecd7ffb0e16f30e88d43c2ba475ce8f6e0f0d4755dfb131 WatchSource:0}: Error finding container fd62ec883d1e08954ecd7ffb0e16f30e88d43c2ba475ce8f6e0f0d4755dfb131: Status 404 returned error can't find the container with id fd62ec883d1e08954ecd7ffb0e16f30e88d43c2ba475ce8f6e0f0d4755dfb131 Nov 03 01:04:47.352078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:47.352036 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" event={"ID":"136d4313-d15e-4903-b21a-c0d4bfed3610","Type":"ContainerStarted","Data":"0f0b6315c3e9bda6fce78eafe5ab046cb23143e4157228adea830e285cfe3292"} Nov 03 01:04:47.352563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:47.352109 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" event={"ID":"136d4313-d15e-4903-b21a-c0d4bfed3610","Type":"ContainerStarted","Data":"fd62ec883d1e08954ecd7ffb0e16f30e88d43c2ba475ce8f6e0f0d4755dfb131"} Nov 03 01:04:47.352563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:47.352137 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:04:47.466069 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:47.465998 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" podStartSLOduration=2.465977017 podStartE2EDuration="2.465977017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:04:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:04:47.464686033 +0000 UTC m=+3577.610380512" watchObservedRunningTime="2026-11-03 01:04:47.465977017 +0000 UTC m=+3577.611671473" Nov 03 01:04:53.314764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:53.314724 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:04:58.365497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:04:58.365464 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:05:03.313725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:03.313677 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:05:10.830088 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:10.830052 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:05:10.845016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:10.844988 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:05:12.162498 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:12.162459 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:05:13.312818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:13.312779 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:05:23.312653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:23.312598 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:05:33.312622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:33.312582 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:05:37.088417 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.088385 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:05:37.088417 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.088416 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:05:37.088942 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.088428 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:37.088942 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.088486 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:05:37.58846595 +0000 UTC m=+3627.734160384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:37.595260 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.595226 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:05:37.595260 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.595257 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:05:37.595260 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.595265 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:37.595513 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:37.595325 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:05:38.595311451 +0000 UTC m=+3628.741005898 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:38.610025 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:38.609989 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:05:38.610025 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:38.610017 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:05:38.610025 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:38.610029 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:38.610599 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:38.610091 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:05:40.610072748 +0000 UTC m=+3630.755767187 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:40.637246 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:40.637216 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:05:40.637246 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:40.637240 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:05:40.637246 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:40.637250 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:40.637729 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:40.637301 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:05:44.637287615 +0000 UTC m=+3634.782982062 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:42.157506 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:42.157459 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:05:43.312745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:43.312709 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:05:44.693617 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:44.693585 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:05:44.693617 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:44.693611 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:05:44.693617 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:44.693621 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:44.694107 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:44.693698 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:05:52.693685316 +0000 UTC m=+3642.839379762 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:52.703511 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:52.703468 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:05:52.703511 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:52.703507 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:05:52.703511 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:52.703518 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:52.704018 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:05:52.703587 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:06:08.703564267 +0000 UTC m=+3658.849258718 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:05:53.312977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:05:53.312938 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:03.312353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:03.312316 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:08.733926 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:08.733888 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:06:08.733926 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:08.733914 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:06:08.733926 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:08.733923 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:06:08.734672 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:08.733981 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:06:40.733968721 +0000 UTC m=+3690.879663168 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:06:09.994793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:09.994752 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:12.165880 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:12.165839 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:06:19.995319 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:19.995280 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:29.995255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:29.995213 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:33.576294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.576263 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq"] Nov 03 01:06:33.581023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.581005 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.696049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.696003 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq"] Nov 03 01:06:33.699971 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.699938 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b47e01-openshift-default-dockercfg-h5jt2\"" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706124 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706776 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706828 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706853 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706887 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706911 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.706954 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.707021 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9gbq\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.720657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.707069 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.810697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.810651 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9gbq\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.811033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.811012 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.811230 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.811215 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.811430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.811407 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.811596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.811576 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.811742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.811716 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.811908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.811885 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.812039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.812018 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.812260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.812190 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.813277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.813220 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.813794 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.813709 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.814405 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.814356 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.814758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.814737 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.814934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.812053 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.828932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.822552 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.828932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.824556 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.868235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.867868 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9gbq\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.875758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.875737 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token\") pod \"e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:33.895614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:33.895589 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:34.329657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:34.329600 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq"] Nov 03 01:06:34.330685 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:06:34.330610 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fd32d5_fc7d_4de9_bcdb_879b4eb1b3fd.slice/crio-8f8f1ca1914b8358b5daef9674dfa72d946df275e265e4e42aac263251e7787f WatchSource:0}: Error finding container 8f8f1ca1914b8358b5daef9674dfa72d946df275e265e4e42aac263251e7787f: Status 404 returned error can't find the container with id 8f8f1ca1914b8358b5daef9674dfa72d946df275e265e4e42aac263251e7787f Nov 03 01:06:34.333330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:34.333301 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:06:34.333425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:34.333365 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:06:34.333425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:34.333394 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:06:35.337103 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:35.337068 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" event={"ID":"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd","Type":"ContainerStarted","Data":"9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7"} Nov 03 01:06:35.337532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:35.337121 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" event={"ID":"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd","Type":"ContainerStarted","Data":"8f8f1ca1914b8358b5daef9674dfa72d946df275e265e4e42aac263251e7787f"} Nov 03 01:06:35.498119 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:35.498060 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podStartSLOduration=2.498042798 podStartE2EDuration="2.498042798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:35.490893419 +0000 UTC m=+3685.636588003" watchObservedRunningTime="2026-11-03 01:06:35.498042798 +0000 UTC m=+3685.643737257" Nov 03 01:06:35.896574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:35.896541 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:35.902775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:35.902748 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:36.345704 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:36.345670 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:36.346573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:36.346554 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:06:39.994672 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:39.994621 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:40.829549 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:40.829516 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:06:40.829549 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:40.829549 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:06:40.829799 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:40.829561 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:06:40.829799 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:40.829652 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:07:44.829621895 +0000 UTC m=+3754.975316342 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:06:42.155536 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:06:42.155501 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:06:43.293734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.293698 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq"] Nov 03 01:06:43.298403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.298381 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.357668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.357642 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq"] Nov 03 01:06:43.365892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.365866 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.366089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.366061 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6z66\" (UniqueName: \"kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66\") pod \"payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.391475 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.391426 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b47e01\"" Nov 03 01:06:43.408866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.408837 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b47e01-dockercfg-rmlcc\"" Nov 03 01:06:43.443484 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.443434 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng"] Nov 03 01:06:43.469652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.468017 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.469652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.468167 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6z66\" (UniqueName: \"kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66\") pod \"payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.469652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.469533 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.470891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.470004 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.532827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.532785 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng"] Nov 03 01:06:43.536452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.536425 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6z66\" (UniqueName: \"kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66\") pod \"payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.570215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.570140 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.570215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.570205 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnrxk\" (UniqueName: \"kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk\") pod \"payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.613806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.613775 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:43.672801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.672576 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.672801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.672669 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnrxk\" (UniqueName: \"kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk\") pod \"payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.673605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.673575 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.742715 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.741410 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnrxk\" (UniqueName: \"kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk\") pod \"payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.818715 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.818687 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:43.997828 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:06:43.997801 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27ba03f1_dc98_4c18_a67b_f554a21829be.slice/crio-7b5f606ebafcf92ed8ac499cb86ac4998b2a3d50055bcc9803f0ad8361915a96 WatchSource:0}: Error finding container 7b5f606ebafcf92ed8ac499cb86ac4998b2a3d50055bcc9803f0ad8361915a96: Status 404 returned error can't find the container with id 7b5f606ebafcf92ed8ac499cb86ac4998b2a3d50055bcc9803f0ad8361915a96 Nov 03 01:06:43.999541 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:43.999520 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq"] Nov 03 01:06:44.209496 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.209465 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng"] Nov 03 01:06:44.209739 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:06:44.209715 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2caee61a_6ff0_49e3_a368_6f08ee8bfeb6.slice/crio-9186695713f9b2aa719f50ef3b30a595e2e479ab4ca30bfd24ccc0c63857c1a1 WatchSource:0}: Error finding container 9186695713f9b2aa719f50ef3b30a595e2e479ab4ca30bfd24ccc0c63857c1a1: Status 404 returned error can't find the container with id 9186695713f9b2aa719f50ef3b30a595e2e479ab4ca30bfd24ccc0c63857c1a1 Nov 03 01:06:44.442662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.442515 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" event={"ID":"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6","Type":"ContainerStarted","Data":"193424159f55dbeab8451208691e6abf167b0f81b27e94a3803392a69ae4375e"} Nov 03 01:06:44.442662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.442581 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" event={"ID":"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6","Type":"ContainerStarted","Data":"9186695713f9b2aa719f50ef3b30a595e2e479ab4ca30bfd24ccc0c63857c1a1"} Nov 03 01:06:44.442662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.442611 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:44.444612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.444580 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" event={"ID":"27ba03f1-dc98-4c18-a67b-f554a21829be","Type":"ContainerStarted","Data":"af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6"} Nov 03 01:06:44.444785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.444664 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" event={"ID":"27ba03f1-dc98-4c18-a67b-f554a21829be","Type":"ContainerStarted","Data":"7b5f606ebafcf92ed8ac499cb86ac4998b2a3d50055bcc9803f0ad8361915a96"} Nov 03 01:06:44.444785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.444693 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:44.564095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.564038 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" podStartSLOduration=1.564019517 podStartE2EDuration="1.564019517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:44.559561559 +0000 UTC m=+3694.705256032" watchObservedRunningTime="2026-11-03 01:06:44.564019517 +0000 UTC m=+3694.709713973" Nov 03 01:06:44.676316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:44.676251 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" podStartSLOduration=1.6762299729999999 podStartE2EDuration="1.676229973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:06:44.673057332 +0000 UTC m=+3694.818751870" watchObservedRunningTime="2026-11-03 01:06:44.676229973 +0000 UTC m=+3694.821924441" Nov 03 01:06:49.995311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:49.995269 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:06:55.457077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:55.457047 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:06:55.457685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:55.457570 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:06:59.352881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:59.352781 5144 patch_prober.go:29] interesting pod/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:06:59.353404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:59.352926 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:06:59.994770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:06:59.994725 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:07:01.734099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:01.734061 5144 generic.go:356] "Generic (PLEG): container finished" podID="098684db-b723-478b-a1a4-3907bd821719" containerID="26e375e1ef3df8e8bd2feb3fa19076041e46abc643d900a82dc8c8f2bba47af8" exitCode=0 Nov 03 01:07:01.734717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:01.734130 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" event={"ID":"098684db-b723-478b-a1a4-3907bd821719","Type":"ContainerDied","Data":"26e375e1ef3df8e8bd2feb3fa19076041e46abc643d900a82dc8c8f2bba47af8"} Nov 03 01:07:01.734717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:01.734599 5144 scope.go:122] "RemoveContainer" containerID="26e375e1ef3df8e8bd2feb3fa19076041e46abc643d900a82dc8c8f2bba47af8" Nov 03 01:07:01.736524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:01.736495 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:07:01.736617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:01.736570 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:07:01.736617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:01.736602 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:07:02.744510 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:02.744469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" event={"ID":"098684db-b723-478b-a1a4-3907bd821719","Type":"ContainerStarted","Data":"831a89146d43268a7cd8acf775146751b5cc5b47fcaaaa37b7a1c8f00192deb1"} Nov 03 01:07:03.281982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:03.281946 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:07:03.282165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:03.282125 5144 patch_prober.go:29] interesting pod/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:07:03.282211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:03.282168 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:07:04.281579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:04.281546 5144 patch_prober.go:29] interesting pod/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:07:04.282050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:04.281607 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:07:05.281793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:05.281764 5144 patch_prober.go:29] interesting pod/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:07:05.282223 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:05.281810 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:07:06.282579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:06.282543 5144 patch_prober.go:29] interesting pod/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:07:06.283008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:06.282597 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:07:07.281921 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:07.281884 5144 patch_prober.go:29] interesting pod/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" start-of-body= Nov 03 01:07:07.282146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:07.281949 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.51:15021/healthz/ready\": dial tcp 10.132.1.51:15021: connect: connection refused" Nov 03 01:07:08.286102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:08.286068 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:07:08.286584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:08.286548 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:07:08.287178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:08.287158 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:07:09.996245 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:09.996211 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:07:12.157682 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:12.157615 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:07:19.994744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:19.994659 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:07:20.467878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:20.467845 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp"] Nov 03 01:07:20.577146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:20.576595 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg"] Nov 03 01:07:25.469168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.469133 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" podUID="f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" containerName="payload-processing" containerID="cri-o://bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24" gracePeriod=25 Nov 03 01:07:25.580102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.579886 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" podUID="e86bfd71-23b3-43a4-aecc-306e7edcd7d3" containerName="payload-pre-processing" containerID="cri-o://67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c" gracePeriod=25 Nov 03 01:07:25.778210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.778184 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:07:25.858511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.858486 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:07:25.949511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.949477 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume\") pod \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " Nov 03 01:07:25.949743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.949534 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb\" (UniqueName: \"kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb\") pod \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\" (UID: \"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6\") " Nov 03 01:07:25.949743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.949696 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume\") pod \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " Nov 03 01:07:25.949888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.949846 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq\" (UniqueName: \"kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq\") pod \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\" (UID: \"e86bfd71-23b3-43a4-aecc-306e7edcd7d3\") " Nov 03 01:07:25.949888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.949876 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume" pod "f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" (UID: "f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:25.950097 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.950070 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume" pod "e86bfd71-23b3-43a4-aecc-306e7edcd7d3" (UID: "e86bfd71-23b3-43a4-aecc-306e7edcd7d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:25.950449 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.950430 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:25.950523 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.950452 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:25.951833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.951815 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb" pod "f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" (UID: "f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6"). InnerVolumeSpecName "kube-api-access-jdvhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:25.951940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:25.951924 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq" pod "e86bfd71-23b3-43a4-aecc-306e7edcd7d3" (UID: "e86bfd71-23b3-43a4-aecc-306e7edcd7d3"). InnerVolumeSpecName "kube-api-access-t7ffq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:26.010107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.010074 5144 generic.go:356] "Generic (PLEG): container finished" podID="f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" containerID="bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24" exitCode=0 Nov 03 01:07:26.010424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.010400 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" event={"ID":"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6","Type":"ContainerDied","Data":"bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24"} Nov 03 01:07:26.010543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.010454 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" event={"ID":"f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6","Type":"ContainerDied","Data":"eb0656ff9304cf3b1b413b0d796e58c52839f01010051fd2a716242374ddc8a5"} Nov 03 01:07:26.010543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.010483 5144 scope.go:122] "RemoveContainer" containerID="bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24" Nov 03 01:07:26.010707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.010645 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp" Nov 03 01:07:26.013289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.013260 5144 generic.go:356] "Generic (PLEG): container finished" podID="e86bfd71-23b3-43a4-aecc-306e7edcd7d3" containerID="67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c" exitCode=0 Nov 03 01:07:26.013394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.013367 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" event={"ID":"e86bfd71-23b3-43a4-aecc-306e7edcd7d3","Type":"ContainerDied","Data":"67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c"} Nov 03 01:07:26.013453 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.013402 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" event={"ID":"e86bfd71-23b3-43a4-aecc-306e7edcd7d3","Type":"ContainerDied","Data":"c657e28b160c5ae6fb0fa3687a559f8b15ae1f9d0c67dafd7545d2c197759001"} Nov 03 01:07:26.013513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.013459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg" Nov 03 01:07:26.024365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.024341 5144 scope.go:122] "RemoveContainer" containerID="bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24" Nov 03 01:07:26.024732 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:26.024709 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24\": container with ID starting with bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24 not found: ID does not exist" containerID="bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24" Nov 03 01:07:26.024828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.024741 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24"} err="failed to get container status \"bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24\": rpc error: code = NotFound desc = could not find container \"bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24\": container with ID starting with bcbeaeaf2362b19f05b2d57bbc63cdd8da0a18f06e7b9bd8dcbc39428f930b24 not found: ID does not exist" Nov 03 01:07:26.024828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.024765 5144 scope.go:122] "RemoveContainer" containerID="67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c" Nov 03 01:07:26.036373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.036351 5144 scope.go:122] "RemoveContainer" containerID="67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c" Nov 03 01:07:26.036683 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:26.036663 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c\": container with ID starting with 67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c not found: ID does not exist" containerID="67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c" Nov 03 01:07:26.036758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.036689 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c"} err="failed to get container status \"67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c\": rpc error: code = NotFound desc = could not find container \"67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c\": container with ID starting with 67795b8d9694996f60ae98c81eb245d048745a9f489cbd755a9f7a1585eab70c not found: ID does not exist" Nov 03 01:07:26.052656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.052614 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7ffq\" (UniqueName: \"kubernetes.io/projected/e86bfd71-23b3-43a4-aecc-306e7edcd7d3-kube-api-access-t7ffq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:26.052746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.052663 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdvhb\" (UniqueName: \"kubernetes.io/projected/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6-kube-api-access-jdvhb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:26.174888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.174867 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp"] Nov 03 01:07:26.178626 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.178604 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cac9f3-667dc897d5-844dp"] Nov 03 01:07:26.286876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.286819 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg"] Nov 03 01:07:26.296689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.296657 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cac9f3-769c4b4dff-wgjbg"] Nov 03 01:07:26.989795 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:26.989761 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:07:27.997608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:27.997574 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86bfd71-23b3-43a4-aecc-306e7edcd7d3" path="/var/lib/kubelet/pods/e86bfd71-23b3-43a4-aecc-306e7edcd7d3/volumes" Nov 03 01:07:27.998008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:27.997986 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" path="/var/lib/kubelet/pods/f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6/volumes" Nov 03 01:07:36.989943 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:36.989902 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:07:42.155800 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:42.155764 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:07:44.893657 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:44.892762 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9b79b389f9e6: secret "agc-cred-9b79b389f9e6" not found Nov 03 01:07:44.893657 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:44.892794 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-e535ec6795f9: secret "agc-cred-e535ec6795f9" not found Nov 03 01:07:44.893657 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:44.892806 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-68495976f-8hfk4: [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:07:44.893657 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:07:44.892875 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials podName:372f1149-53d4-4317-b340-b0c6ec57b2ff nodeName:}" failed. No retries permitted until 2026-11-03 01:09:46.892855986 +0000 UTC m=+3877.038550438 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials") pod "payload-processing-external-model-68495976f-8hfk4" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff") : [secret "agc-cred-9b79b389f9e6" not found, secret "agc-cred-e535ec6795f9" not found] Nov 03 01:07:46.989897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:46.989864 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:07:48.048996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:48.048961 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4"] Nov 03 01:07:53.049807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.049768 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" podUID="136d4313-d15e-4903-b21a-c0d4bfed3610" containerName="payload-pre-processing" containerID="cri-o://0f0b6315c3e9bda6fce78eafe5ab046cb23143e4157228adea830e285cfe3292" gracePeriod=25 Nov 03 01:07:53.234779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.234741 5144 generic.go:356] "Generic (PLEG): container finished" podID="136d4313-d15e-4903-b21a-c0d4bfed3610" containerID="0f0b6315c3e9bda6fce78eafe5ab046cb23143e4157228adea830e285cfe3292" exitCode=0 Nov 03 01:07:53.235087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.235062 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" event={"ID":"136d4313-d15e-4903-b21a-c0d4bfed3610","Type":"ContainerDied","Data":"0f0b6315c3e9bda6fce78eafe5ab046cb23143e4157228adea830e285cfe3292"} Nov 03 01:07:53.349348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.349322 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:07:53.411450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.411422 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume\") pod \"136d4313-d15e-4903-b21a-c0d4bfed3610\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " Nov 03 01:07:53.411604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.411590 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47\" (UniqueName: \"kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47\") pod \"136d4313-d15e-4903-b21a-c0d4bfed3610\" (UID: \"136d4313-d15e-4903-b21a-c0d4bfed3610\") " Nov 03 01:07:53.411813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.411791 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume" pod "136d4313-d15e-4903-b21a-c0d4bfed3610" (UID: "136d4313-d15e-4903-b21a-c0d4bfed3610"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:07:53.412141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.412124 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/136d4313-d15e-4903-b21a-c0d4bfed3610-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:53.413866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.413845 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47" pod "136d4313-d15e-4903-b21a-c0d4bfed3610" (UID: "136d4313-d15e-4903-b21a-c0d4bfed3610"). InnerVolumeSpecName "kube-api-access-gkj47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:07:53.513553 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:53.513527 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkj47\" (UniqueName: \"kubernetes.io/projected/136d4313-d15e-4903-b21a-c0d4bfed3610-kube-api-access-gkj47\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:07:54.244544 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:54.244506 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" event={"ID":"136d4313-d15e-4903-b21a-c0d4bfed3610","Type":"ContainerDied","Data":"fd62ec883d1e08954ecd7ffb0e16f30e88d43c2ba475ce8f6e0f0d4755dfb131"} Nov 03 01:07:54.245042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:54.244590 5144 scope.go:122] "RemoveContainer" containerID="0f0b6315c3e9bda6fce78eafe5ab046cb23143e4157228adea830e285cfe3292" Nov 03 01:07:54.245042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:54.244523 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4" Nov 03 01:07:54.414420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:54.414387 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4"] Nov 03 01:07:54.418246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:54.418218 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-67b917-6d476846b7-zqlz4"] Nov 03 01:07:55.998414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:55.998379 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136d4313-d15e-4903-b21a-c0d4bfed3610" path="/var/lib/kubelet/pods/136d4313-d15e-4903-b21a-c0d4bfed3610/volumes" Nov 03 01:07:56.990205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:07:56.990168 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:08:06.990118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:06.990083 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:08:12.159822 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:08:12.159777 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:08:13.441174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:13.441140 5144 scope.go:122] "RemoveContainer" containerID="3e270989d59701a626fd610aee0dc4f5078a55ff520c6696ca436ee8b477fda6" Nov 03 01:08:16.990301 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:16.990255 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:08:26.990218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:26.990180 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:08:34.989931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:34.989884 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:08:36.348224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.348191 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:36.348729 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.348250 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:36.651171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.651068 5144 generic.go:356] "Generic (PLEG): container finished" podID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerID="9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7" exitCode=0 Nov 03 01:08:36.651171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.651151 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" event={"ID":"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd","Type":"ContainerDied","Data":"9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7"} Nov 03 01:08:36.651765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.651742 5144 scope.go:122] "RemoveContainer" containerID="9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7" Nov 03 01:08:36.653848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.653817 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:08:36.653976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.653892 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:08:36.653976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:36.653933 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:08:37.663964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:37.663923 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" event={"ID":"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd","Type":"ContainerStarted","Data":"1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e"} Nov 03 01:08:37.778654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:37.776052 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89"] Nov 03 01:08:37.778654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:37.776307 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" containerID="cri-o://d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb" gracePeriod=30 Nov 03 01:08:37.896853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:37.896809 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:08:37.897175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:37.897151 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:37.897229 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:37.897210 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:38.896421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:38.896384 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:38.896930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:38.896452 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:39.896304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:39.896269 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:39.896518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:39.896338 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:40.896809 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:40.896772 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:40.897306 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:40.896854 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:41.896394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:41.896356 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:41.896666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:41.896418 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:42.171887 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:08:42.171790 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:08:42.896457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:42.896421 5144 patch_prober.go:29] interesting pod/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" start-of-body= Nov 03 01:08:42.896728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:42.896481 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.54:15021/healthz/ready\": dial tcp 10.132.1.54:15021: connect: connection refused" Nov 03 01:08:43.121436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.121406 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:08:43.220984 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.220923 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221062 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221093 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221180 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221219 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221233 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221322 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221359 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221459 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221581 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:08:43.221974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221623 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs\") pod \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\" (UID: \"9006f61a-46ec-4dbb-94b0-afca485bd4e2\") " Nov 03 01:08:43.221974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.221935 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:43.222151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222040 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:43.222151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222096 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:43.222837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222659 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.222837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222689 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.222837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222704 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.222837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222719 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.222837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.222732 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.224525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.224500 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:43.224835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.224798 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:08:43.225064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.225046 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "kube-api-access-z66rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:08:43.225148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.225096 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo" pod "9006f61a-46ec-4dbb-94b0-afca485bd4e2" (UID: "9006f61a-46ec-4dbb-94b0-afca485bd4e2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:08:43.324359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.324318 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.324359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.324355 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66rt\" (UniqueName: \"kubernetes.io/projected/9006f61a-46ec-4dbb-94b0-afca485bd4e2-kube-api-access-z66rt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.324359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.324367 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.324670 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.324376 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9006f61a-46ec-4dbb-94b0-afca485bd4e2-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:08:43.719558 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.719521 5144 generic.go:356] "Generic (PLEG): container finished" podID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerID="d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb" exitCode=0 Nov 03 01:08:43.719801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.719577 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" Nov 03 01:08:43.719801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.719603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" event={"ID":"9006f61a-46ec-4dbb-94b0-afca485bd4e2","Type":"ContainerDied","Data":"d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb"} Nov 03 01:08:43.719801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.719690 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89" event={"ID":"9006f61a-46ec-4dbb-94b0-afca485bd4e2","Type":"ContainerDied","Data":"70d365c11b5ae638dcd95d2a6c395416a30d5857bbfb0a3f7eccbef045616ae7"} Nov 03 01:08:43.719801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.719721 5144 scope.go:122] "RemoveContainer" containerID="d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb" Nov 03 01:08:43.735298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.735275 5144 scope.go:122] "RemoveContainer" containerID="bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3" Nov 03 01:08:43.746444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.746418 5144 scope.go:122] "RemoveContainer" containerID="d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb" Nov 03 01:08:43.746817 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:08:43.746794 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb\": container with ID starting with d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb not found: ID does not exist" containerID="d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb" Nov 03 01:08:43.746901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.746828 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb"} err="failed to get container status \"d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb\": rpc error: code = NotFound desc = could not find container \"d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb\": container with ID starting with d2e2bcad1914a14f0d54cb1da342b56ec0d2db89fc290b84bd21a450a1055aeb not found: ID does not exist" Nov 03 01:08:43.746901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.746847 5144 scope.go:122] "RemoveContainer" containerID="bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3" Nov 03 01:08:43.747149 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:08:43.747127 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3\": container with ID starting with bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3 not found: ID does not exist" containerID="bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3" Nov 03 01:08:43.747194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.747158 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3"} err="failed to get container status \"bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3\": rpc error: code = NotFound desc = could not find container \"bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3\": container with ID starting with bec6ab27f50c52a3fb622cb59802b4502da6181b0c510de0268c42c9e72f67c3 not found: ID does not exist" Nov 03 01:08:43.885998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.885966 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89"] Nov 03 01:08:43.902287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.902253 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:08:43.902722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.902696 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:08:43.903459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.903441 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:08:43.936092 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.936057 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-cac9f3-openshift-default-76656fd585-47f89"] Nov 03 01:08:43.999341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:43.999255 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" path="/var/lib/kubelet/pods/9006f61a-46ec-4dbb-94b0-afca485bd4e2/volumes" Nov 03 01:08:44.990148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:44.990113 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:08:54.989688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:08:54.989616 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:09:04.989517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:04.989475 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:09:12.156333 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:12.156295 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:09:14.989720 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:14.989675 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:09:16.809663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:16.809618 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk"] Nov 03 01:09:16.810133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:16.809847 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" containerID="cri-o://831a89146d43268a7cd8acf775146751b5cc5b47fcaaaa37b7a1c8f00192deb1" gracePeriod=30 Nov 03 01:09:21.041451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.041413 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw"] Nov 03 01:09:21.042126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042112 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" containerName="payload-processing" Nov 03 01:09:21.042174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042126 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" containerName="payload-processing" Nov 03 01:09:21.042174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042151 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e86bfd71-23b3-43a4-aecc-306e7edcd7d3" containerName="payload-pre-processing" Nov 03 01:09:21.042174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042155 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86bfd71-23b3-43a4-aecc-306e7edcd7d3" containerName="payload-pre-processing" Nov 03 01:09:21.042174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042174 5144 cpu_manager.go:414] "containerMap: removing container" podUID="136d4313-d15e-4903-b21a-c0d4bfed3610" containerName="payload-pre-processing" Nov 03 01:09:21.042309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042179 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136d4313-d15e-4903-b21a-c0d4bfed3610" containerName="payload-pre-processing" Nov 03 01:09:21.042309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042196 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" Nov 03 01:09:21.042309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042201 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" Nov 03 01:09:21.042309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042219 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" Nov 03 01:09:21.042309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042223 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" Nov 03 01:09:21.042478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042315 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86bfd71-23b3-43a4-aecc-306e7edcd7d3" containerName="payload-pre-processing" Nov 03 01:09:21.042478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042327 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="136d4313-d15e-4903-b21a-c0d4bfed3610" containerName="payload-pre-processing" Nov 03 01:09:21.042478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042336 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" Nov 03 01:09:21.042478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042354 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9006f61a-46ec-4dbb-94b0-afca485bd4e2" containerName="istio-proxy" Nov 03 01:09:21.042478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.042361 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ffa9a8-3e55-4b34-bd4b-fb133117a2f6" containerName="payload-processing" Nov 03 01:09:21.045327 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.045311 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.072907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.072884 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.073097 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.073076 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.073239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.073224 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m6sv\" (UniqueName: \"kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.101881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.101854 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw"] Nov 03 01:09:21.145783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.145737 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:09:21.145904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.145795 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:09:21.174529 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.174501 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.174665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.174590 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m6sv\" (UniqueName: \"kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.174798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.174777 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.175233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.175210 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.177228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.177206 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.231723 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.231701 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m6sv\" (UniqueName: \"kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv\") pod \"maas-api-revoke-sub-6e076d9d-bdldw\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.360527 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.360453 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:21.737060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:21.737029 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw"] Nov 03 01:09:21.737753 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:09:21.737726 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569876da_e1c8_48d5_83fb_116a19a4af1a.slice/crio-08eb3b90b8c99217f66cabc585a84a9d64772b6b3fb82ee56ea5bfac1f61d5f5 WatchSource:0}: Error finding container 08eb3b90b8c99217f66cabc585a84a9d64772b6b3fb82ee56ea5bfac1f61d5f5: Status 404 returned error can't find the container with id 08eb3b90b8c99217f66cabc585a84a9d64772b6b3fb82ee56ea5bfac1f61d5f5 Nov 03 01:09:22.068923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.068883 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" event={"ID":"569876da-e1c8-48d5-83fb-116a19a4af1a","Type":"ContainerStarted","Data":"08eb3b90b8c99217f66cabc585a84a9d64772b6b3fb82ee56ea5bfac1f61d5f5"} Nov 03 01:09:22.071397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.071359 5144 generic.go:356] "Generic (PLEG): container finished" podID="098684db-b723-478b-a1a4-3907bd821719" containerID="831a89146d43268a7cd8acf775146751b5cc5b47fcaaaa37b7a1c8f00192deb1" exitCode=0 Nov 03 01:09:22.071550 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.071410 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" event={"ID":"098684db-b723-478b-a1a4-3907bd821719","Type":"ContainerDied","Data":"831a89146d43268a7cd8acf775146751b5cc5b47fcaaaa37b7a1c8f00192deb1"} Nov 03 01:09:22.071550 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.071465 5144 scope.go:122] "RemoveContainer" containerID="26e375e1ef3df8e8bd2feb3fa19076041e46abc643d900a82dc8c8f2bba47af8" Nov 03 01:09:22.081908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.081884 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:09:22.188998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.188971 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189071 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189168 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189253 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189217 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189301 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189359 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189414 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189477 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.189581 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189553 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:22.190284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189728 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo\") pod \"098684db-b723-478b-a1a4-3907bd821719\" (UID: \"098684db-b723-478b-a1a4-3907bd821719\") " Nov 03 01:09:22.190284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189887 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:22.190284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.189954 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:22.190284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.190213 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:22.190601 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.190574 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:22.190753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.190732 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.190826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.190761 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/098684db-b723-478b-a1a4-3907bd821719-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.190826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.190776 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.190826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.190805 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.192037 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.192017 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "kube-api-access-78zbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:22.192894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.192868 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:09:22.193251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.193228 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:09:22.193314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.193267 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token" pod "098684db-b723-478b-a1a4-3907bd821719" (UID: "098684db-b723-478b-a1a4-3907bd821719"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:22.292355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.292322 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.292497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.292358 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/098684db-b723-478b-a1a4-3907bd821719-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.292497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.292372 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/098684db-b723-478b-a1a4-3907bd821719-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.292497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.292388 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78zbq\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-kube-api-access-78zbq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:22.292497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:22.292402 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/098684db-b723-478b-a1a4-3907bd821719-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:23.080722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.080684 5144 generic.go:356] "Generic (PLEG): container finished" podID="569876da-e1c8-48d5-83fb-116a19a4af1a" containerID="b9a10c1c0f20ea89ae133636b37393e04e5c07da664e12b395259b6993e75c07" exitCode=0 Nov 03 01:09:23.081125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.080759 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" event={"ID":"569876da-e1c8-48d5-83fb-116a19a4af1a","Type":"ContainerDied","Data":"b9a10c1c0f20ea89ae133636b37393e04e5c07da664e12b395259b6993e75c07"} Nov 03 01:09:23.082488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.082468 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" Nov 03 01:09:23.082611 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.082471 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk" event={"ID":"098684db-b723-478b-a1a4-3907bd821719","Type":"ContainerDied","Data":"7016388292e3465edf8b30f5b5cfee42aa265a45e169325655b1f4fd29b94732"} Nov 03 01:09:23.082611 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.082578 5144 scope.go:122] "RemoveContainer" containerID="831a89146d43268a7cd8acf775146751b5cc5b47fcaaaa37b7a1c8f00192deb1" Nov 03 01:09:23.370684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.370579 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk"] Nov 03 01:09:23.376650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.376523 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-244924-openshift-default-84f6ff94bf-w8vbk"] Nov 03 01:09:23.997500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:23.997465 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="098684db-b723-478b-a1a4-3907bd821719" path="/var/lib/kubelet/pods/098684db-b723-478b-a1a4-3907bd821719/volumes" Nov 03 01:09:24.243425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.243404 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:24.314620 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.314593 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca\") pod \"569876da-e1c8-48d5-83fb-116a19a4af1a\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " Nov 03 01:09:24.314815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.314797 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv\" (UniqueName: \"kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv\") pod \"569876da-e1c8-48d5-83fb-116a19a4af1a\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " Nov 03 01:09:24.314934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.314913 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca" pod "569876da-e1c8-48d5-83fb-116a19a4af1a" (UID: "569876da-e1c8-48d5-83fb-116a19a4af1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:24.314979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.314936 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca\") pod \"569876da-e1c8-48d5-83fb-116a19a4af1a\" (UID: \"569876da-e1c8-48d5-83fb-116a19a4af1a\") " Nov 03 01:09:24.315491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.315410 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:24.317063 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.317042 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv" pod "569876da-e1c8-48d5-83fb-116a19a4af1a" (UID: "569876da-e1c8-48d5-83fb-116a19a4af1a"). InnerVolumeSpecName "kube-api-access-5m6sv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:24.317136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.317111 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca" pod "569876da-e1c8-48d5-83fb-116a19a4af1a" (UID: "569876da-e1c8-48d5-83fb-116a19a4af1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:09:24.416334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.416301 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569876da-e1c8-48d5-83fb-116a19a4af1a-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:24.416334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.416330 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m6sv\" (UniqueName: \"kubernetes.io/projected/569876da-e1c8-48d5-83fb-116a19a4af1a-kube-api-access-5m6sv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:24.990135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:24.990101 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:09:25.102603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:25.102572 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" Nov 03 01:09:25.102603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:25.102599 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw" event={"ID":"569876da-e1c8-48d5-83fb-116a19a4af1a","Type":"ContainerDied","Data":"08eb3b90b8c99217f66cabc585a84a9d64772b6b3fb82ee56ea5bfac1f61d5f5"} Nov 03 01:09:25.102835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:25.102664 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08eb3b90b8c99217f66cabc585a84a9d64772b6b3fb82ee56ea5bfac1f61d5f5" Nov 03 01:09:27.048640 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:27.048605 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w"] Nov 03 01:09:27.095739 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:27.095713 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d1353522-cjq9w"] Nov 03 01:09:27.997094 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:27.997059 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03237d99-29ab-4d00-b0a5-065ce884bc6e" path="/var/lib/kubelet/pods/03237d99-29ab-4d00-b0a5-065ce884bc6e/volumes" Nov 03 01:09:28.046114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:28.046081 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895"] Nov 03 01:09:28.095149 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:28.095122 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5694f383-gz895"] Nov 03 01:09:29.997264 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:29.997229 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3" path="/var/lib/kubelet/pods/c5bcbff5-8926-47ae-b7e9-7a8cccec0cf3/volumes" Nov 03 01:09:33.635896 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:33.635863 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68495976f-8hfk4"] Nov 03 01:09:33.687027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:33.686994 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn"] Nov 03 01:09:33.688464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:33.688424 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:09:38.636793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:38.636749 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" podUID="372f1149-53d4-4317-b340-b0c6ec57b2ff" containerName="payload-processing" containerID="cri-o://898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab" gracePeriod=25 Nov 03 01:09:38.688174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:38.688087 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" containerID="cri-o://863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a" gracePeriod=25 Nov 03 01:09:39.003794 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.003772 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:09:39.008391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.008368 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:09:39.109535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109509 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials\") pod \"372f1149-53d4-4317-b340-b0c6ec57b2ff\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " Nov 03 01:09:39.109774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109558 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume\") pod \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " Nov 03 01:09:39.109774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109579 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume\") pod \"372f1149-53d4-4317-b340-b0c6ec57b2ff\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " Nov 03 01:09:39.109774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109624 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials\") pod \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " Nov 03 01:09:39.109774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109669 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl\") pod \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " Nov 03 01:09:39.109774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109722 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume\") pod \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\" (UID: \"2f17efb3-775c-496c-9a35-4a1f1abaf9e9\") " Nov 03 01:09:39.110089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109778 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume\") pod \"372f1149-53d4-4317-b340-b0c6ec57b2ff\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " Nov 03 01:09:39.110089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109969 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume" pod "2f17efb3-775c-496c-9a35-4a1f1abaf9e9" (UID: "2f17efb3-775c-496c-9a35-4a1f1abaf9e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:39.110089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.109975 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68\") pod \"372f1149-53d4-4317-b340-b0c6ec57b2ff\" (UID: \"372f1149-53d4-4317-b340-b0c6ec57b2ff\") " Nov 03 01:09:39.110089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.110022 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume" pod "372f1149-53d4-4317-b340-b0c6ec57b2ff" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:39.110453 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.110430 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume" pod "372f1149-53d4-4317-b340-b0c6ec57b2ff" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:39.110575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.110548 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume" pod "2f17efb3-775c-496c-9a35-4a1f1abaf9e9" (UID: "2f17efb3-775c-496c-9a35-4a1f1abaf9e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:39.110683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.110653 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.110683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.110675 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.110813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.110689 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/372f1149-53d4-4317-b340-b0c6ec57b2ff-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.112122 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.112096 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials" pod "372f1149-53d4-4317-b340-b0c6ec57b2ff" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:39.112258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.112176 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68" pod "372f1149-53d4-4317-b340-b0c6ec57b2ff" (UID: "372f1149-53d4-4317-b340-b0c6ec57b2ff"). InnerVolumeSpecName "kube-api-access-pbj68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:39.112817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.112788 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials" pod "2f17efb3-775c-496c-9a35-4a1f1abaf9e9" (UID: "2f17efb3-775c-496c-9a35-4a1f1abaf9e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:39.112817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.112804 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl" pod "2f17efb3-775c-496c-9a35-4a1f1abaf9e9" (UID: "2f17efb3-775c-496c-9a35-4a1f1abaf9e9"). InnerVolumeSpecName "kube-api-access-wptpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:39.211928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.211853 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbj68\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-kube-api-access-pbj68\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.211928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.211880 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/372f1149-53d4-4317-b340-b0c6ec57b2ff-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.211928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.211890 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.211928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.211899 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wptpl\" (UniqueName: \"kubernetes.io/projected/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-kube-api-access-wptpl\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.211928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.211908 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f17efb3-775c-496c-9a35-4a1f1abaf9e9-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:39.236080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.236044 5144 generic.go:356] "Generic (PLEG): container finished" podID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerID="863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a" exitCode=0 Nov 03 01:09:39.236249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.236122 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" Nov 03 01:09:39.236249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.236164 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" event={"ID":"2f17efb3-775c-496c-9a35-4a1f1abaf9e9","Type":"ContainerDied","Data":"863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a"} Nov 03 01:09:39.236249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.236212 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn" event={"ID":"2f17efb3-775c-496c-9a35-4a1f1abaf9e9","Type":"ContainerDied","Data":"79b8c0ba11b1a68ad143ed0641e83bfd94860abef77a3369ce9d21ad2f62fa2e"} Nov 03 01:09:39.236249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.236241 5144 scope.go:122] "RemoveContainer" containerID="863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a" Nov 03 01:09:39.238494 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.238465 5144 generic.go:356] "Generic (PLEG): container finished" podID="372f1149-53d4-4317-b340-b0c6ec57b2ff" containerID="898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab" exitCode=0 Nov 03 01:09:39.238612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.238562 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" event={"ID":"372f1149-53d4-4317-b340-b0c6ec57b2ff","Type":"ContainerDied","Data":"898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab"} Nov 03 01:09:39.238612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.238601 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" event={"ID":"372f1149-53d4-4317-b340-b0c6ec57b2ff","Type":"ContainerDied","Data":"b0771c37a5e30e9d52c3a614da9460cc46b28fe1947011fc98785f1ed42dd985"} Nov 03 01:09:39.238801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.238784 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-68495976f-8hfk4" Nov 03 01:09:39.266873 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.266850 5144 scope.go:122] "RemoveContainer" containerID="863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a" Nov 03 01:09:39.267256 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:39.267227 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a\": container with ID starting with 863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a not found: ID does not exist" containerID="863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a" Nov 03 01:09:39.267376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.267265 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a"} err="failed to get container status \"863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a\": rpc error: code = NotFound desc = could not find container \"863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a\": container with ID starting with 863beb21ff70ceb983ddf2a242dfdf6d3d732747b11338b903a83f741c840b7a not found: ID does not exist" Nov 03 01:09:39.267376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.267292 5144 scope.go:122] "RemoveContainer" containerID="898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab" Nov 03 01:09:39.280355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.280339 5144 scope.go:122] "RemoveContainer" containerID="898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab" Nov 03 01:09:39.280681 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:39.280616 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab\": container with ID starting with 898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab not found: ID does not exist" containerID="898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab" Nov 03 01:09:39.280744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.280676 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab"} err="failed to get container status \"898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab\": rpc error: code = NotFound desc = could not find container \"898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab\": container with ID starting with 898d89006216c7569009180b078b85c2a6d14e45cdcfb4d0b9586a453eec5cab not found: ID does not exist" Nov 03 01:09:39.400393 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.400352 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-68495976f-8hfk4"] Nov 03 01:09:39.404050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.404024 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-68495976f-8hfk4"] Nov 03 01:09:39.510400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.510370 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn"] Nov 03 01:09:39.513926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:39.513903 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-6d78f968dc-mhwhn"] Nov 03 01:09:40.000096 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:40.000017 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" path="/var/lib/kubelet/pods/2f17efb3-775c-496c-9a35-4a1f1abaf9e9/volumes" Nov 03 01:09:40.000572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:40.000556 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372f1149-53d4-4317-b340-b0c6ec57b2ff" path="/var/lib/kubelet/pods/372f1149-53d4-4317-b340-b0c6ec57b2ff/volumes" Nov 03 01:09:42.157547 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:42.157512 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:09:43.564387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.564358 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-q5tr9"] Nov 03 01:09:43.565057 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565043 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" Nov 03 01:09:43.565109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565058 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" Nov 03 01:09:43.565109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565087 5144 cpu_manager.go:414] "containerMap: removing container" podUID="569876da-e1c8-48d5-83fb-116a19a4af1a" containerName="revoke-keys" Nov 03 01:09:43.565109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565092 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569876da-e1c8-48d5-83fb-116a19a4af1a" containerName="revoke-keys" Nov 03 01:09:43.565109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565101 5144 cpu_manager.go:414] "containerMap: removing container" podUID="372f1149-53d4-4317-b340-b0c6ec57b2ff" containerName="payload-processing" Nov 03 01:09:43.565109 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565106 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372f1149-53d4-4317-b340-b0c6ec57b2ff" containerName="payload-processing" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565117 5144 cpu_manager.go:414] "containerMap: removing container" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565121 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565134 5144 cpu_manager.go:414] "containerMap: removing container" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565139 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565236 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565249 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f17efb3-775c-496c-9a35-4a1f1abaf9e9" containerName="payload-processing" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565260 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="569876da-e1c8-48d5-83fb-116a19a4af1a" containerName="revoke-keys" Nov 03 01:09:43.565270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.565267 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="372f1149-53d4-4317-b340-b0c6ec57b2ff" containerName="payload-processing" Nov 03 01:09:43.570047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.570027 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.621982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.621957 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:09:43.622133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.622062 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:09:43.624254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.624232 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-q5tr9"] Nov 03 01:09:43.669363 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.669327 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:09:43.669363 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.669351 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:09:43.669524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.669368 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9ceb1d2c2705\"" Nov 03 01:09:43.669524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.669408 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-mz2fr\"" Nov 03 01:09:43.672261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.672238 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.672352 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.672292 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.672571 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.672549 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.672609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.672580 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.712264 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.712212 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh"] Nov 03 01:09:43.773978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.773944 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.774176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.773987 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.774176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.774056 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.774176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.774102 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.774389 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:43.774270 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:43.774389 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:43.774290 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-q5tr9: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:43.774779 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:43.774759 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials podName:8423eeb1-8adc-459a-9bd5-f18e5f87606f nodeName:}" failed. No retries permitted until 2026-11-03 01:09:44.274729131 +0000 UTC m=+3874.420423580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials") pod "payload-processing-external-model-7c76876988-q5tr9" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:43.774924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.774847 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.774924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.774872 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:43.824561 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.824475 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76"] Nov 03 01:09:43.836372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:43.836337 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:44.282760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:44.281543 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:44.282760 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:44.281952 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:44.282760 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:44.281973 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-q5tr9: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:44.282760 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:44.282046 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials podName:8423eeb1-8adc-459a-9bd5-f18e5f87606f nodeName:}" failed. No retries permitted until 2026-11-03 01:09:45.282027774 +0000 UTC m=+3875.427722226 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials") pod "payload-processing-external-model-7c76876988-q5tr9" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:45.296156 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:45.296121 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:45.296801 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:45.296307 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:45.296801 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:45.296331 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-q5tr9: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:45.296801 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:45.296423 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials podName:8423eeb1-8adc-459a-9bd5-f18e5f87606f nodeName:}" failed. No retries permitted until 2026-11-03 01:09:47.296401603 +0000 UTC m=+3877.442096059 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials") pod "payload-processing-external-model-7c76876988-q5tr9" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:47.321777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:47.321741 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials\") pod \"payload-processing-external-model-7c76876988-q5tr9\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:47.322194 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:47.321879 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:47.322194 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:47.321896 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-q5tr9: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:47.322194 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:47.321952 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials podName:8423eeb1-8adc-459a-9bd5-f18e5f87606f nodeName:}" failed. No retries permitted until 2026-11-03 01:09:51.32193899 +0000 UTC m=+3881.467633424 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials") pod "payload-processing-external-model-7c76876988-q5tr9" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:48.591184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:48.591146 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-q5tr9"] Nov 03 01:09:48.591589 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:48.591480 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" podUID="8423eeb1-8adc-459a-9bd5-f18e5f87606f" Nov 03 01:09:48.713436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:48.713383 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" podUID="0350ec1b-16ff-47ec-acbe-f666ecc05ef9" containerName="payload-processing" containerID="cri-o://0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c" gracePeriod=25 Nov 03 01:09:48.825744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:48.825687 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" podUID="e8395528-f715-4eb3-acc4-b98ceaa553a0" containerName="payload-pre-processing" containerID="cri-o://3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9" gracePeriod=25 Nov 03 01:09:49.092415 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.089340 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:09:49.160371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.160292 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume\") pod \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " Nov 03 01:09:49.160525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.160391 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx\" (UniqueName: \"kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx\") pod \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\" (UID: \"0350ec1b-16ff-47ec-acbe-f666ecc05ef9\") " Nov 03 01:09:49.161098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.161066 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume" pod "0350ec1b-16ff-47ec-acbe-f666ecc05ef9" (UID: "0350ec1b-16ff-47ec-acbe-f666ecc05ef9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:49.162954 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.162929 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx" pod "0350ec1b-16ff-47ec-acbe-f666ecc05ef9" (UID: "0350ec1b-16ff-47ec-acbe-f666ecc05ef9"). InnerVolumeSpecName "kube-api-access-tkbjx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:49.172112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.172093 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:09:49.283658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.278748 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.283658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.278778 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tkbjx\" (UniqueName: \"kubernetes.io/projected/0350ec1b-16ff-47ec-acbe-f666ecc05ef9-kube-api-access-tkbjx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.329330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.329293 5144 generic.go:356] "Generic (PLEG): container finished" podID="e8395528-f715-4eb3-acc4-b98ceaa553a0" containerID="3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9" exitCode=0 Nov 03 01:09:49.329501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.329340 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" Nov 03 01:09:49.329501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.329387 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" event={"ID":"e8395528-f715-4eb3-acc4-b98ceaa553a0","Type":"ContainerDied","Data":"3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9"} Nov 03 01:09:49.329501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.329453 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76" event={"ID":"e8395528-f715-4eb3-acc4-b98ceaa553a0","Type":"ContainerDied","Data":"038541b2082c221038183fde6b2691931efbd0d5635d39b945a94b7944b86813"} Nov 03 01:09:49.329501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.329484 5144 scope.go:122] "RemoveContainer" containerID="3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9" Nov 03 01:09:49.330907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.330881 5144 generic.go:356] "Generic (PLEG): container finished" podID="0350ec1b-16ff-47ec-acbe-f666ecc05ef9" containerID="0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c" exitCode=0 Nov 03 01:09:49.331055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.330938 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" Nov 03 01:09:49.331055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.330964 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" event={"ID":"0350ec1b-16ff-47ec-acbe-f666ecc05ef9","Type":"ContainerDied","Data":"0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c"} Nov 03 01:09:49.331055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.331029 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh" event={"ID":"0350ec1b-16ff-47ec-acbe-f666ecc05ef9","Type":"ContainerDied","Data":"88e179f980d2301c0ec3ef14af63042b271c3792dbf26d5d1fdda190c48aaca1"} Nov 03 01:09:49.331208 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.331104 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:49.347618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.347595 5144 scope.go:122] "RemoveContainer" containerID="3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9" Nov 03 01:09:49.351098 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:49.351071 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9\": container with ID starting with 3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9 not found: ID does not exist" containerID="3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9" Nov 03 01:09:49.351236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.351109 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9"} err="failed to get container status \"3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9\": rpc error: code = NotFound desc = could not find container \"3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9\": container with ID starting with 3034bf2e92e602d784633b861815a40ce1e64adad239ef03f648ad71968f39a9 not found: ID does not exist" Nov 03 01:09:49.351236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.351132 5144 scope.go:122] "RemoveContainer" containerID="0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c" Nov 03 01:09:49.351834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.351814 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:49.366127 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.366108 5144 scope.go:122] "RemoveContainer" containerID="0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c" Nov 03 01:09:49.366498 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:49.366474 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c\": container with ID starting with 0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c not found: ID does not exist" containerID="0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c" Nov 03 01:09:49.366597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.366503 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c"} err="failed to get container status \"0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c\": rpc error: code = NotFound desc = could not find container \"0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c\": container with ID starting with 0457466ac08a845bfcb85104852360525691a3683ec3d8a96e434f077378cb8c not found: ID does not exist" Nov 03 01:09:49.379730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.379699 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn\" (UniqueName: \"kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn\") pod \"e8395528-f715-4eb3-acc4-b98ceaa553a0\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " Nov 03 01:09:49.379848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.379751 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume\") pod \"e8395528-f715-4eb3-acc4-b98ceaa553a0\" (UID: \"e8395528-f715-4eb3-acc4-b98ceaa553a0\") " Nov 03 01:09:49.379848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.379780 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv\") pod \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " Nov 03 01:09:49.380077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380056 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume\") pod \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " Nov 03 01:09:49.380176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380102 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume\") pod \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\" (UID: \"8423eeb1-8adc-459a-9bd5-f18e5f87606f\") " Nov 03 01:09:49.380243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380196 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume" pod "e8395528-f715-4eb3-acc4-b98ceaa553a0" (UID: "e8395528-f715-4eb3-acc4-b98ceaa553a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:49.380702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380676 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume" pod "8423eeb1-8adc-459a-9bd5-f18e5f87606f" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:49.380822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380701 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume" pod "8423eeb1-8adc-459a-9bd5-f18e5f87606f" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:09:49.380944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380922 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.381011 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380951 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8423eeb1-8adc-459a-9bd5-f18e5f87606f-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.381011 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.380966 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8395528-f715-4eb3-acc4-b98ceaa553a0-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.382089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.382069 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn" pod "e8395528-f715-4eb3-acc4-b98ceaa553a0" (UID: "e8395528-f715-4eb3-acc4-b98ceaa553a0"). InnerVolumeSpecName "kube-api-access-pbvjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:49.383101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.383080 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv" pod "8423eeb1-8adc-459a-9bd5-f18e5f87606f" (UID: "8423eeb1-8adc-459a-9bd5-f18e5f87606f"). InnerVolumeSpecName "kube-api-access-m5pgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:09:49.482714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.481758 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5pgv\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-kube-api-access-m5pgv\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.482714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.481795 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvjn\" (UniqueName: \"kubernetes.io/projected/e8395528-f715-4eb3-acc4-b98ceaa553a0-kube-api-access-pbvjn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:49.808436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.808401 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh"] Nov 03 01:09:49.814457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.814430 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-4db6f5-6f8d6d4f75-2fkbh"] Nov 03 01:09:49.922986 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.922923 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76"] Nov 03 01:09:49.928884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:49.928857 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-4db6f5-7bbcd8d867-8dl76"] Nov 03 01:09:50.006189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:50.006161 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0350ec1b-16ff-47ec-acbe-f666ecc05ef9" path="/var/lib/kubelet/pods/0350ec1b-16ff-47ec-acbe-f666ecc05ef9/volumes" Nov 03 01:09:50.006604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:50.006589 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8395528-f715-4eb3-acc4-b98ceaa553a0" path="/var/lib/kubelet/pods/e8395528-f715-4eb3-acc4-b98ceaa553a0/volumes" Nov 03 01:09:50.341811 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:50.341729 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-q5tr9" Nov 03 01:09:50.566111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:50.566078 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-q5tr9"] Nov 03 01:09:50.569594 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:50.569572 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-q5tr9"] Nov 03 01:09:50.612094 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:50.612008 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8423eeb1-8adc-459a-9bd5-f18e5f87606f-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:09:51.996944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:51.996915 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8423eeb1-8adc-459a-9bd5-f18e5f87606f" path="/var/lib/kubelet/pods/8423eeb1-8adc-459a-9bd5-f18e5f87606f/volumes" Nov 03 01:09:53.783749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.783716 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-9mxkn"] Nov 03 01:09:53.784395 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784381 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0350ec1b-16ff-47ec-acbe-f666ecc05ef9" containerName="payload-processing" Nov 03 01:09:53.784395 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784396 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0350ec1b-16ff-47ec-acbe-f666ecc05ef9" containerName="payload-processing" Nov 03 01:09:53.784494 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784444 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e8395528-f715-4eb3-acc4-b98ceaa553a0" containerName="payload-pre-processing" Nov 03 01:09:53.784494 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784450 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8395528-f715-4eb3-acc4-b98ceaa553a0" containerName="payload-pre-processing" Nov 03 01:09:53.784581 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784572 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="098684db-b723-478b-a1a4-3907bd821719" containerName="istio-proxy" Nov 03 01:09:53.784618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784586 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0350ec1b-16ff-47ec-acbe-f666ecc05ef9" containerName="payload-processing" Nov 03 01:09:53.784618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.784596 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8395528-f715-4eb3-acc4-b98ceaa553a0" containerName="payload-pre-processing" Nov 03 01:09:53.799121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.799093 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.846391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.846361 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-9mxkn"] Nov 03 01:09:53.851574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.851547 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b89ff\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.851745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.851606 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.851799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.851761 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.851835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.851804 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.890786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.890758 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:09:53.891090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.891005 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-9ceb1d2c2705\"" Nov 03 01:09:53.891262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.891050 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-9jnps\"" Nov 03 01:09:53.891262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.891098 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:09:53.891437 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.891149 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:09:53.900507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.900491 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:09:53.953741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.953706 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b89ff\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.953865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.953775 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.953865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.953837 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.954013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.953877 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.954582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.954557 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.954763 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.954741 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:53.956881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:53.956853 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:54.009750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:54.009721 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b89ff\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff\") pod \"payload-processing-external-model-7c76876988-9mxkn\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:54.114372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:54.114282 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:54.511200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:54.511172 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-9mxkn"] Nov 03 01:09:54.511977 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:09:54.511943 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5c10a5_847d_4f17_b400_b7a02a362702.slice/crio-d6593fdebdd63225c412234351452725a9ad0044f483f4c20787aecc36f5df06 WatchSource:0}: Error finding container d6593fdebdd63225c412234351452725a9ad0044f483f4c20787aecc36f5df06: Status 404 returned error can't find the container with id d6593fdebdd63225c412234351452725a9ad0044f483f4c20787aecc36f5df06 Nov 03 01:09:55.393781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:55.393745 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" event={"ID":"4a5c10a5-847d-4f17-b400-b7a02a362702","Type":"ContainerStarted","Data":"074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10"} Nov 03 01:09:55.394283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:55.393803 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" event={"ID":"4a5c10a5-847d-4f17-b400-b7a02a362702","Type":"ContainerStarted","Data":"d6593fdebdd63225c412234351452725a9ad0044f483f4c20787aecc36f5df06"} Nov 03 01:09:55.398001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:55.394618 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:09:55.475516 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:55.475477 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:55.475516 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:55.475509 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-9mxkn: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:55.475862 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:55.475586 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:09:55.97556589 +0000 UTC m=+3886.121260344 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:55.516688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:55.515156 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" podStartSLOduration=2.515134309 podStartE2EDuration="2.515134309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:09:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:09:55.511871643 +0000 UTC m=+3885.657566121" watchObservedRunningTime="2026-11-03 01:09:55.515134309 +0000 UTC m=+3885.660828767" Nov 03 01:09:55.982969 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:55.982938 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:55.982969 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:55.982964 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-9mxkn: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:55.983204 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:55.983016 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:09:56.983002282 +0000 UTC m=+3887.128696729 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:56.606897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:56.606863 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-9mxkn"] Nov 03 01:09:56.995879 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:56.995802 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:56.995879 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:56.995831 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-9mxkn: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:56.996088 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:56.995904 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:09:58.995891241 +0000 UTC m=+3889.141585676 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:57.417936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:09:57.417901 5144 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" secret="" err="secret \"payload-processing-external-model-dockercfg-9jnps\" not found" Nov 03 01:09:57.509766 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:57.509734 5144 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 01:09:57.510057 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:57.510040 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:09:58.010017924 +0000 UTC m=+3888.155712371 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : configmap "payload-processing-external-model-plugins" not found Nov 03 01:09:58.016625 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:58.016587 5144 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 01:09:58.017139 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:58.016726 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:09:59.016703237 +0000 UTC m=+3889.162397677 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : configmap "payload-processing-external-model-plugins" not found Nov 03 01:09:59.042906 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:59.042011 5144 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 01:09:59.042906 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:59.042099 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:10:01.042079064 +0000 UTC m=+3891.187773512 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : configmap "payload-processing-external-model-plugins" not found Nov 03 01:09:59.042906 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:59.042522 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:59.042906 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:59.042540 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-9mxkn: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:09:59.042906 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:09:59.042588 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:10:03.042571042 +0000 UTC m=+3893.188265555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:10:00.958658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:00.957747 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd"] Nov 03 01:10:00.968320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:00.968288 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.000573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.000102 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.000573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.000151 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpfrw\" (UniqueName: \"kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.000573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.000219 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.042657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.034579 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd"] Nov 03 01:10:01.076279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.076163 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:10:01.076279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.076243 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:10:01.188258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.188220 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.188530 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.188506 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpfrw\" (UniqueName: \"kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.188722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.188703 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.189271 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:01.189248 5144 configmap.go:193] Couldn't get configMap models-as-a-service/payload-processing-external-model-plugins: configmap "payload-processing-external-model-plugins" not found Nov 03 01:10:01.189454 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:01.189441 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:10:05.18941938 +0000 UTC m=+3895.335113839 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : configmap "payload-processing-external-model-plugins" not found Nov 03 01:10:01.207756 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.192093 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.207756 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.194068 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.287535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.287484 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpfrw\" (UniqueName: \"kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw\") pod \"maas-api-revoke-sub-0b5cda05-skjzd\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.305427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.302199 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:01.717646 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:10:01.717598 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffc3ff9e_725c_4eb3_9f25_a88a55f1819a.slice/crio-e93cc5f5be693fb08f219d1152a58247cb36aba04c27c12665dba8b613e15018 WatchSource:0}: Error finding container e93cc5f5be693fb08f219d1152a58247cb36aba04c27c12665dba8b613e15018: Status 404 returned error can't find the container with id e93cc5f5be693fb08f219d1152a58247cb36aba04c27c12665dba8b613e15018 Nov 03 01:10:01.717940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:01.717914 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd"] Nov 03 01:10:02.517676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:02.517618 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" event={"ID":"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a","Type":"ContainerStarted","Data":"538e427cfe45f6b1c01a96d8c3516bdc95607cd1dbfd50fb6cbec98d185c7d7d"} Nov 03 01:10:02.518181 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:02.517704 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" event={"ID":"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a","Type":"ContainerStarted","Data":"e93cc5f5be693fb08f219d1152a58247cb36aba04c27c12665dba8b613e15018"} Nov 03 01:10:02.633184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:02.633125 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" podStartSLOduration=2.633112889 podStartE2EDuration="2.633112889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:10:02.630961415 +0000 UTC m=+3892.776655875" watchObservedRunningTime="2026-11-03 01:10:02.633112889 +0000 UTC m=+3892.778807344" Nov 03 01:10:03.112293 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:03.112261 5144 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-9ceb1d2c2705: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:10:03.112293 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:03.112287 5144 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7c76876988-9mxkn: secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:10:03.112515 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:03.112341 5144 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials podName:4a5c10a5-847d-4f17-b400-b7a02a362702 nodeName:}" failed. No retries permitted until 2026-11-03 01:10:11.112326971 +0000 UTC m=+3901.258021420 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials") pod "payload-processing-external-model-7c76876988-9mxkn" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702") : secret "agc-cred-9ceb1d2c2705" not found Nov 03 01:10:03.425163 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.425070 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" podUID="4a5c10a5-847d-4f17-b400-b7a02a362702" containerName="payload-processing" containerID="cri-o://074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10" gracePeriod=25 Nov 03 01:10:03.528051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.528017 5144 generic.go:356] "Generic (PLEG): container finished" podID="ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" containerID="538e427cfe45f6b1c01a96d8c3516bdc95607cd1dbfd50fb6cbec98d185c7d7d" exitCode=0 Nov 03 01:10:03.528459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.528105 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" event={"ID":"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a","Type":"ContainerDied","Data":"538e427cfe45f6b1c01a96d8c3516bdc95607cd1dbfd50fb6cbec98d185c7d7d"} Nov 03 01:10:03.804668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.801847 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:10:03.820392 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.820354 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials\") pod \"4a5c10a5-847d-4f17-b400-b7a02a362702\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " Nov 03 01:10:03.820584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.820478 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume\") pod \"4a5c10a5-847d-4f17-b400-b7a02a362702\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " Nov 03 01:10:03.820584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.820528 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff\") pod \"4a5c10a5-847d-4f17-b400-b7a02a362702\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " Nov 03 01:10:03.820763 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.820601 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume\") pod \"4a5c10a5-847d-4f17-b400-b7a02a362702\" (UID: \"4a5c10a5-847d-4f17-b400-b7a02a362702\") " Nov 03 01:10:03.821135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.821105 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume" pod "4a5c10a5-847d-4f17-b400-b7a02a362702" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:03.821347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.821327 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume" pod "4a5c10a5-847d-4f17-b400-b7a02a362702" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:03.823648 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.823605 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials" pod "4a5c10a5-847d-4f17-b400-b7a02a362702" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:03.824049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.824030 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff" pod "4a5c10a5-847d-4f17-b400-b7a02a362702" (UID: "4a5c10a5-847d-4f17-b400-b7a02a362702"). InnerVolumeSpecName "kube-api-access-b89ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:03.922964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.922929 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:03.922964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.922964 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:03.923743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.922979 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b89ff\" (UniqueName: \"kubernetes.io/projected/4a5c10a5-847d-4f17-b400-b7a02a362702-kube-api-access-b89ff\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:03.923743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:03.922994 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4a5c10a5-847d-4f17-b400-b7a02a362702-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:04.537457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.537425 5144 generic.go:356] "Generic (PLEG): container finished" podID="4a5c10a5-847d-4f17-b400-b7a02a362702" containerID="074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10" exitCode=0 Nov 03 01:10:04.537977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.537475 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" Nov 03 01:10:04.537977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.537510 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" event={"ID":"4a5c10a5-847d-4f17-b400-b7a02a362702","Type":"ContainerDied","Data":"074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10"} Nov 03 01:10:04.537977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.537575 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7c76876988-9mxkn" event={"ID":"4a5c10a5-847d-4f17-b400-b7a02a362702","Type":"ContainerDied","Data":"d6593fdebdd63225c412234351452725a9ad0044f483f4c20787aecc36f5df06"} Nov 03 01:10:04.537977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.537610 5144 scope.go:122] "RemoveContainer" containerID="074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10" Nov 03 01:10:04.549768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.549748 5144 scope.go:122] "RemoveContainer" containerID="074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10" Nov 03 01:10:04.550085 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:04.550064 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10\": container with ID starting with 074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10 not found: ID does not exist" containerID="074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10" Nov 03 01:10:04.550160 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.550100 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10"} err="failed to get container status \"074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10\": rpc error: code = NotFound desc = could not find container \"074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10\": container with ID starting with 074de9c7f07f6395c0739a4971f05e00768c4cda12de64a672e265bc7c73fc10 not found: ID does not exist" Nov 03 01:10:04.703809 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.703777 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-9mxkn"] Nov 03 01:10:04.721114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.721090 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:04.737569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.737546 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca\") pod \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " Nov 03 01:10:04.737767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.737751 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw\" (UniqueName: \"kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw\") pod \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " Nov 03 01:10:04.737848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.737786 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca\") pod \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\" (UID: \"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a\") " Nov 03 01:10:04.738310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.738250 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca" pod "ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" (UID: "ffc3ff9e-725c-4eb3-9f25-a88a55f1819a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:10:04.738544 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.738401 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:04.739946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.739920 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca" pod "ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" (UID: "ffc3ff9e-725c-4eb3-9f25-a88a55f1819a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:10:04.740089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.740067 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw" pod "ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" (UID: "ffc3ff9e-725c-4eb3-9f25-a88a55f1819a"). InnerVolumeSpecName "kube-api-access-zpfrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:10:04.753361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.753338 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7c76876988-9mxkn"] Nov 03 01:10:04.839430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.839356 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpfrw\" (UniqueName: \"kubernetes.io/projected/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-kube-api-access-zpfrw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:04.839430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:04.839384 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:10:05.547568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:05.547532 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" Nov 03 01:10:05.547568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:05.547540 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd" event={"ID":"ffc3ff9e-725c-4eb3-9f25-a88a55f1819a","Type":"ContainerDied","Data":"e93cc5f5be693fb08f219d1152a58247cb36aba04c27c12665dba8b613e15018"} Nov 03 01:10:05.548075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:05.547604 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e93cc5f5be693fb08f219d1152a58247cb36aba04c27c12665dba8b613e15018" Nov 03 01:10:06.017654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:06.012277 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5c10a5-847d-4f17-b400-b7a02a362702" path="/var/lib/kubelet/pods/4a5c10a5-847d-4f17-b400-b7a02a362702/volumes" Nov 03 01:10:10.993425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:10.993394 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:10:10.994230 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:10.994205 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:10:12.162495 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:12.162443 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:10:42.158599 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:10:42.158502 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:10:45.297752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:10:45.297704 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:11:00.946359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:00.946316 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng"] Nov 03 01:11:01.058266 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:01.058235 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq"] Nov 03 01:11:05.946711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:05.946626 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" podUID="2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" containerName="payload-processing" containerID="cri-o://193424159f55dbeab8451208691e6abf167b0f81b27e94a3803392a69ae4375e" gracePeriod=25 Nov 03 01:11:06.059462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.059427 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" podUID="27ba03f1-dc98-4c18-a67b-f554a21829be" containerName="payload-pre-processing" containerID="cri-o://af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6" gracePeriod=25 Nov 03 01:11:06.071393 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.071367 5144 generic.go:356] "Generic (PLEG): container finished" podID="2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" containerID="193424159f55dbeab8451208691e6abf167b0f81b27e94a3803392a69ae4375e" exitCode=0 Nov 03 01:11:06.071514 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.071437 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" event={"ID":"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6","Type":"ContainerDied","Data":"193424159f55dbeab8451208691e6abf167b0f81b27e94a3803392a69ae4375e"} Nov 03 01:11:06.229797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.229171 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:11:06.316649 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.316606 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume\") pod \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " Nov 03 01:11:06.316850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.316832 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk\" (UniqueName: \"kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk\") pod \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\" (UID: \"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6\") " Nov 03 01:11:06.316994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.316962 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume" pod "2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" (UID: "2caee61a-6ff0-49e3-a368-6f08ee8bfeb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:06.317392 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.317371 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:06.319185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.319158 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk" pod "2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" (UID: "2caee61a-6ff0-49e3-a368-6f08ee8bfeb6"). InnerVolumeSpecName "kube-api-access-nnrxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:06.343654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.343304 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:11:06.418838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.418814 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume\") pod \"27ba03f1-dc98-4c18-a67b-f554a21829be\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " Nov 03 01:11:06.419018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.418854 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66\" (UniqueName: \"kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66\") pod \"27ba03f1-dc98-4c18-a67b-f554a21829be\" (UID: \"27ba03f1-dc98-4c18-a67b-f554a21829be\") " Nov 03 01:11:06.419241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.419218 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume" pod "27ba03f1-dc98-4c18-a67b-f554a21829be" (UID: "27ba03f1-dc98-4c18-a67b-f554a21829be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:06.419241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.419235 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnrxk\" (UniqueName: \"kubernetes.io/projected/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6-kube-api-access-nnrxk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:06.421094 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.421073 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66" pod "27ba03f1-dc98-4c18-a67b-f554a21829be" (UID: "27ba03f1-dc98-4c18-a67b-f554a21829be"). InnerVolumeSpecName "kube-api-access-x6z66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:06.520586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.520557 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/27ba03f1-dc98-4c18-a67b-f554a21829be-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:06.520586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:06.520586 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6z66\" (UniqueName: \"kubernetes.io/projected/27ba03f1-dc98-4c18-a67b-f554a21829be-kube-api-access-x6z66\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:07.082581 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.082544 5144 generic.go:356] "Generic (PLEG): container finished" podID="27ba03f1-dc98-4c18-a67b-f554a21829be" containerID="af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6" exitCode=0 Nov 03 01:11:07.083065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.082601 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" Nov 03 01:11:07.083065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.082645 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" event={"ID":"27ba03f1-dc98-4c18-a67b-f554a21829be","Type":"ContainerDied","Data":"af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6"} Nov 03 01:11:07.083065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.082713 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq" event={"ID":"27ba03f1-dc98-4c18-a67b-f554a21829be","Type":"ContainerDied","Data":"7b5f606ebafcf92ed8ac499cb86ac4998b2a3d50055bcc9803f0ad8361915a96"} Nov 03 01:11:07.083065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.082743 5144 scope.go:122] "RemoveContainer" containerID="af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6" Nov 03 01:11:07.084782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.084755 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" event={"ID":"2caee61a-6ff0-49e3-a368-6f08ee8bfeb6","Type":"ContainerDied","Data":"9186695713f9b2aa719f50ef3b30a595e2e479ab4ca30bfd24ccc0c63857c1a1"} Nov 03 01:11:07.084912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.084849 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng" Nov 03 01:11:07.095423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.095407 5144 scope.go:122] "RemoveContainer" containerID="af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6" Nov 03 01:11:07.095736 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:11:07.095712 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6\": container with ID starting with af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6 not found: ID does not exist" containerID="af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6" Nov 03 01:11:07.095833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.095742 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6"} err="failed to get container status \"af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6\": rpc error: code = NotFound desc = could not find container \"af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6\": container with ID starting with af3ea8600002d015d58410f51437517684854ee9022448789a613c24a2c64cd6 not found: ID does not exist" Nov 03 01:11:07.095833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.095758 5144 scope.go:122] "RemoveContainer" containerID="193424159f55dbeab8451208691e6abf167b0f81b27e94a3803392a69ae4375e" Nov 03 01:11:07.246788 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.246710 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq"] Nov 03 01:11:07.302655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.301188 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b47e01-59cf6c55-m9jmq"] Nov 03 01:11:07.360857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.360830 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng"] Nov 03 01:11:07.365016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.364990 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b47e01-5f4f599665-vjjng"] Nov 03 01:11:07.997543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.997513 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27ba03f1-dc98-4c18-a67b-f554a21829be" path="/var/lib/kubelet/pods/27ba03f1-dc98-4c18-a67b-f554a21829be/volumes" Nov 03 01:11:07.997936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:07.997920 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" path="/var/lib/kubelet/pods/2caee61a-6ff0-49e3-a368-6f08ee8bfeb6/volumes" Nov 03 01:11:12.154109 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:11:12.154073 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:11:13.592620 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:13.592583 5144 scope.go:122] "RemoveContainer" containerID="5027a9f089b2306e4ae1bdb2ceab17aec7cbcd175c89d48099df2f9393531245" Nov 03 01:11:13.605463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:13.605439 5144 scope.go:122] "RemoveContainer" containerID="5c2fb708dd2582db612c7c3f9f5e4d2d8fd0a2bb1cac0f7c07ba16b55721bcaf" Nov 03 01:11:42.155780 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:11:42.155739 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:11:42.659496 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.659447 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q"] Nov 03 01:11:42.660534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660505 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" containerName="payload-processing" Nov 03 01:11:42.660534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660534 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" containerName="payload-processing" Nov 03 01:11:42.660804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660561 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4a5c10a5-847d-4f17-b400-b7a02a362702" containerName="payload-processing" Nov 03 01:11:42.660804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660569 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5c10a5-847d-4f17-b400-b7a02a362702" containerName="payload-processing" Nov 03 01:11:42.660804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660594 5144 cpu_manager.go:414] "containerMap: removing container" podUID="27ba03f1-dc98-4c18-a67b-f554a21829be" containerName="payload-pre-processing" Nov 03 01:11:42.660804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660602 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27ba03f1-dc98-4c18-a67b-f554a21829be" containerName="payload-pre-processing" Nov 03 01:11:42.660804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660652 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" containerName="revoke-keys" Nov 03 01:11:42.660804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660659 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" containerName="revoke-keys" Nov 03 01:11:42.661130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660857 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5c10a5-847d-4f17-b400-b7a02a362702" containerName="payload-processing" Nov 03 01:11:42.661130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660873 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="27ba03f1-dc98-4c18-a67b-f554a21829be" containerName="payload-pre-processing" Nov 03 01:11:42.661130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660887 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" containerName="revoke-keys" Nov 03 01:11:42.661130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.660907 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2caee61a-6ff0-49e3-a368-6f08ee8bfeb6" containerName="payload-processing" Nov 03 01:11:42.668228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.668188 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.720086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.720053 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q"] Nov 03 01:11:42.764047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.764011 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:11:42.764230 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.764169 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:11:42.786045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.786013 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.786196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.786130 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.786354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.786328 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc2l4\" (UniqueName: \"kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.887803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.887770 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.888006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.887834 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.888006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.887941 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc2l4\" (UniqueName: \"kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.888564 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.888534 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.890485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.890457 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.945750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.945679 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc2l4\" (UniqueName: \"kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4\") pod \"maas-api-revoke-sub-dd1f35c3-5g26q\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:42.984936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:42.984897 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:43.369604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:43.369574 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q"] Nov 03 01:11:43.370094 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:11:43.370069 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31a38c1_07e7_46ba_9c2d_753df766b600.slice/crio-41bd245b72f3fa06d34fda8ccf4042ccc1a93a3342436c972c073c6a97734402 WatchSource:0}: Error finding container 41bd245b72f3fa06d34fda8ccf4042ccc1a93a3342436c972c073c6a97734402: Status 404 returned error can't find the container with id 41bd245b72f3fa06d34fda8ccf4042ccc1a93a3342436c972c073c6a97734402 Nov 03 01:11:44.377284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:44.377249 5144 generic.go:356] "Generic (PLEG): container finished" podID="b31a38c1-07e7-46ba-9c2d-753df766b600" containerID="79dd4d79c7cb17e8f58c871940aa6a0a963550497cd5cfd91293ef127b7ad559" exitCode=0 Nov 03 01:11:44.377834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:44.377335 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" event={"ID":"b31a38c1-07e7-46ba-9c2d-753df766b600","Type":"ContainerDied","Data":"79dd4d79c7cb17e8f58c871940aa6a0a963550497cd5cfd91293ef127b7ad559"} Nov 03 01:11:44.377834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:44.377431 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" event={"ID":"b31a38c1-07e7-46ba-9c2d-753df766b600","Type":"ContainerStarted","Data":"41bd245b72f3fa06d34fda8ccf4042ccc1a93a3342436c972c073c6a97734402"} Nov 03 01:11:45.558233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.558199 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:45.724356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.724272 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4\" (UniqueName: \"kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4\") pod \"b31a38c1-07e7-46ba-9c2d-753df766b600\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " Nov 03 01:11:45.724797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.724777 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca\") pod \"b31a38c1-07e7-46ba-9c2d-753df766b600\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " Nov 03 01:11:45.725013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.724998 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca\") pod \"b31a38c1-07e7-46ba-9c2d-753df766b600\" (UID: \"b31a38c1-07e7-46ba-9c2d-753df766b600\") " Nov 03 01:11:45.725978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.725225 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca" pod "b31a38c1-07e7-46ba-9c2d-753df766b600" (UID: "b31a38c1-07e7-46ba-9c2d-753df766b600"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:11:45.730056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.730018 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca" pod "b31a38c1-07e7-46ba-9c2d-753df766b600" (UID: "b31a38c1-07e7-46ba-9c2d-753df766b600"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:11:45.730761 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.730743 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4" pod "b31a38c1-07e7-46ba-9c2d-753df766b600" (UID: "b31a38c1-07e7-46ba-9c2d-753df766b600"). InnerVolumeSpecName "kube-api-access-lc2l4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:11:45.826951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.826910 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:45.826951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.826947 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b31a38c1-07e7-46ba-9c2d-753df766b600-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:45.826951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:45.826957 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc2l4\" (UniqueName: \"kubernetes.io/projected/b31a38c1-07e7-46ba-9c2d-753df766b600-kube-api-access-lc2l4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:11:46.393338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:46.393256 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" Nov 03 01:11:46.393474 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:46.393255 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q" event={"ID":"b31a38c1-07e7-46ba-9c2d-753df766b600","Type":"ContainerDied","Data":"41bd245b72f3fa06d34fda8ccf4042ccc1a93a3342436c972c073c6a97734402"} Nov 03 01:11:46.393474 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:11:46.393383 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="41bd245b72f3fa06d34fda8ccf4042ccc1a93a3342436c972c073c6a97734402" Nov 03 01:12:07.089781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:07.089747 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq"] Nov 03 01:12:07.090409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:07.089971 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" containerID="cri-o://1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e" gracePeriod=30 Nov 03 01:12:12.157295 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:12:12.157252 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:12:12.366106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.365999 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:12:12.482065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482033 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482126 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482150 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482192 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482226 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482255 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482276 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482300 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482343 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy\") pod \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\" (UID: \"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd\") " Nov 03 01:12:12.482608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482357 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:12:12.482879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482695 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:12:12.482879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482708 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:12:12.482879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482716 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:12:12.483108 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.482975 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:12:12.483335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.483226 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.483335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.483249 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.483335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.483265 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.483335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.483280 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.484812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.484776 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:12:12.484932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.484834 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:12:12.484932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.484871 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:12:12.484932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.484892 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq" pod "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" (UID: "17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd"). InnerVolumeSpecName "kube-api-access-z9gbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:12:12.584931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.584899 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9gbq\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-kube-api-access-z9gbq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.584931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.584930 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.585134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.584941 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.585134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.584954 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.585134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.584966 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:12.606989 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.606955 5144 generic.go:356] "Generic (PLEG): container finished" podID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerID="1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e" exitCode=0 Nov 03 01:12:12.607195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.606999 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" Nov 03 01:12:12.607195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.607038 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" event={"ID":"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd","Type":"ContainerDied","Data":"1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e"} Nov 03 01:12:12.607195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.607102 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq" event={"ID":"17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd","Type":"ContainerDied","Data":"8f8f1ca1914b8358b5daef9674dfa72d946df275e265e4e42aac263251e7787f"} Nov 03 01:12:12.607195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.607134 5144 scope.go:122] "RemoveContainer" containerID="1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e" Nov 03 01:12:12.618951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.618932 5144 scope.go:122] "RemoveContainer" containerID="9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7" Nov 03 01:12:12.629593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.629576 5144 scope.go:122] "RemoveContainer" containerID="1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e" Nov 03 01:12:12.629892 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:12:12.629875 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e\": container with ID starting with 1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e not found: ID does not exist" containerID="1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e" Nov 03 01:12:12.629946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.629899 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e"} err="failed to get container status \"1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e\": rpc error: code = NotFound desc = could not find container \"1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e\": container with ID starting with 1c1f7b821c0f319d466f29f1d4291d63111e7f79af11840736f83f3093af6e9e not found: ID does not exist" Nov 03 01:12:12.629946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.629917 5144 scope.go:122] "RemoveContainer" containerID="9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7" Nov 03 01:12:12.630203 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:12:12.630181 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7\": container with ID starting with 9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7 not found: ID does not exist" containerID="9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7" Nov 03 01:12:12.630263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.630212 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7"} err="failed to get container status \"9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7\": rpc error: code = NotFound desc = could not find container \"9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7\": container with ID starting with 9561f0d54354ccb9c407071ff910c79f4fa601a90e5a181c7df00cfa780b7cd7 not found: ID does not exist" Nov 03 01:12:12.771963 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.771931 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq"] Nov 03 01:12:12.775517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:12.775493 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b47e01-openshift-default-7bb4f8c97f-lbspq"] Nov 03 01:12:14.004534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:14.004496 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" path="/var/lib/kubelet/pods/17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd/volumes" Nov 03 01:12:23.009059 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009022 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b"] Nov 03 01:12:23.009699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009684 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b31a38c1-07e7-46ba-9c2d-753df766b600" containerName="revoke-keys" Nov 03 01:12:23.009764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009700 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b31a38c1-07e7-46ba-9c2d-753df766b600" containerName="revoke-keys" Nov 03 01:12:23.009764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009708 5144 cpu_manager.go:414] "containerMap: removing container" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" Nov 03 01:12:23.009764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009714 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" Nov 03 01:12:23.009764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009730 5144 cpu_manager.go:414] "containerMap: removing container" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" Nov 03 01:12:23.009764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009735 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" Nov 03 01:12:23.009930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009865 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" Nov 03 01:12:23.009930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.009882 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b31a38c1-07e7-46ba-9c2d-753df766b600" containerName="revoke-keys" Nov 03 01:12:23.013445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.013430 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.068988 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.068962 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b"] Nov 03 01:12:23.101927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.101898 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjfh\" (UniqueName: \"kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.102079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.101966 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.102421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.102372 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.114012 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.113987 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:12:23.114146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.114084 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:12:23.204359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.204320 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.204645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.204608 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjfh\" (UniqueName: \"kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.204853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.204836 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.205919 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.205896 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.209815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.209043 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.265482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.265408 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjfh\" (UniqueName: \"kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh\") pod \"maas-api-revoke-sub-b032cc2f-sv42b\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.328459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.328428 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:23.706721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:23.706690 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b"] Nov 03 01:12:23.707836 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:12:23.707808 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0aafdda_1fee_4532_aa89_f6af6e1ba6c8.slice/crio-f9ef8db11126018897e60af78aae0d31d965973328e53739562c4ace42f9c948 WatchSource:0}: Error finding container f9ef8db11126018897e60af78aae0d31d965973328e53739562c4ace42f9c948: Status 404 returned error can't find the container with id f9ef8db11126018897e60af78aae0d31d965973328e53739562c4ace42f9c948 Nov 03 01:12:24.703930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:24.703872 5144 generic.go:356] "Generic (PLEG): container finished" podID="c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" containerID="a60491f963258f21b51581de13674b92ab82984f8d321dcc2d59b94fef1d0cab" exitCode=0 Nov 03 01:12:24.703930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:24.703916 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" event={"ID":"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8","Type":"ContainerDied","Data":"a60491f963258f21b51581de13674b92ab82984f8d321dcc2d59b94fef1d0cab"} Nov 03 01:12:24.704430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:24.703983 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" event={"ID":"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8","Type":"ContainerStarted","Data":"f9ef8db11126018897e60af78aae0d31d965973328e53739562c4ace42f9c948"} Nov 03 01:12:25.887288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.887259 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:25.938845 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.938816 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh\" (UniqueName: \"kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh\") pod \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " Nov 03 01:12:25.939014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.938866 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca\") pod \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " Nov 03 01:12:25.939014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.938926 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca\") pod \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\" (UID: \"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8\") " Nov 03 01:12:25.939372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.939337 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca" pod "c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" (UID: "c0aafdda-1fee-4532-aa89-f6af6e1ba6c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:12:25.939663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.939645 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:25.941747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.941724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca" pod "c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" (UID: "c0aafdda-1fee-4532-aa89-f6af6e1ba6c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:12:25.941826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:25.941757 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh" pod "c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" (UID: "c0aafdda-1fee-4532-aa89-f6af6e1ba6c8"). InnerVolumeSpecName "kube-api-access-fqjfh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:12:26.041193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:26.041152 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqjfh\" (UniqueName: \"kubernetes.io/projected/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-kube-api-access-fqjfh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:26.041193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:26.041195 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:12:26.721403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:26.721375 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" Nov 03 01:12:26.721588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:26.721414 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b" event={"ID":"c0aafdda-1fee-4532-aa89-f6af6e1ba6c8","Type":"ContainerDied","Data":"f9ef8db11126018897e60af78aae0d31d965973328e53739562c4ace42f9c948"} Nov 03 01:12:26.721588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:26.721484 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9ef8db11126018897e60af78aae0d31d965973328e53739562c4ace42f9c948" Nov 03 01:12:35.885774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:35.885721 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:35.929674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:35.925702 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:36.028656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:36.012671 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:36.099667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:12:36.086954 5144 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:12:42.156418 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:12:42.156374 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:13:12.158481 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:13:12.158440 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:13:24.693502 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.693466 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj"] Nov 03 01:13:24.694141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.694126 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" containerName="revoke-keys" Nov 03 01:13:24.694186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.694142 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" containerName="revoke-keys" Nov 03 01:13:24.694287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.694278 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="17fd32d5-fc7d-4de9-bcdb-879b4eb1b3fd" containerName="istio-proxy" Nov 03 01:13:24.694341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.694292 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" containerName="revoke-keys" Nov 03 01:13:24.698008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.697984 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.751157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.751122 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-aee0dd37-openshift-default-dockercfg-6x62v\"" Nov 03 01:13:24.806157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.806122 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj"] Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858164 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49pbk\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858258 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858306 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858364 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858408 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858472 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858557 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858615 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.862657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.858658 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.961750 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.961827 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.961852 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.961904 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49pbk\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.961962 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.962011 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.962061 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.962296 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.962386 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.962663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.962446 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.963530 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.963353 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.963652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.963600 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.964286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.964244 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.964491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.964467 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.967030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.966873 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:24.968156 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:24.967567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:25.020168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.020132 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:25.020448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.020423 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49pbk\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk\") pod \"e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:25.312856 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.312816 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:25.748874 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.748843 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj"] Nov 03 01:13:25.749507 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:13:25.749484 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b57094_a4ae_470d_8842_197f9bd4e7c6.slice/crio-edf4af348460a5dfad07d09e488d9699f48c4e8c248dbb974c48f0450523d1e4 WatchSource:0}: Error finding container edf4af348460a5dfad07d09e488d9699f48c4e8c248dbb974c48f0450523d1e4: Status 404 returned error can't find the container with id edf4af348460a5dfad07d09e488d9699f48c4e8c248dbb974c48f0450523d1e4 Nov 03 01:13:25.752218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.752187 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:13:25.752303 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.752256 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:13:25.752303 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:25.752287 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:13:26.369376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:26.369339 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" event={"ID":"d8b57094-a4ae-470d-8842-197f9bd4e7c6","Type":"ContainerStarted","Data":"547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3"} Nov 03 01:13:26.369585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:26.369401 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" event={"ID":"d8b57094-a4ae-470d-8842-197f9bd4e7c6","Type":"ContainerStarted","Data":"edf4af348460a5dfad07d09e488d9699f48c4e8c248dbb974c48f0450523d1e4"} Nov 03 01:13:26.489468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:26.489417 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podStartSLOduration=2.489402725 podStartE2EDuration="2.489402725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:26.486152351 +0000 UTC m=+4096.631846822" watchObservedRunningTime="2026-11-03 01:13:26.489402725 +0000 UTC m=+4096.635097181" Nov 03 01:13:27.313566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:27.313532 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:27.320173 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:27.320142 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:27.377982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:27.377952 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:27.378979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:27.378952 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:13:32.947235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:32.947197 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt"] Nov 03 01:13:32.951938 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:32.951909 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.004124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.004089 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-aee0dd37\"" Nov 03 01:13:33.007757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.007720 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt"] Nov 03 01:13:33.081937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.081900 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-aee0dd37-dockercfg-kn65m\"" Nov 03 01:13:33.086886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.086769 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6t47\" (UniqueName: \"kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47\") pod \"payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.087479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.087446 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.146545 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.146508 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs"] Nov 03 01:13:33.152745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.152718 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.163385 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.163033 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs"] Nov 03 01:13:33.191078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.191040 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6t47\" (UniqueName: \"kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47\") pod \"payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.191274 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.191219 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.191966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.191945 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.248646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.248557 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6t47\" (UniqueName: \"kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47\") pod \"payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.268025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.267983 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:33.292996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.292956 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgrhd\" (UniqueName: \"kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd\") pod \"payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.293177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.293158 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume\") pod \"payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.395002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.394963 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgrhd\" (UniqueName: \"kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd\") pod \"payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.395206 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.395092 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume\") pod \"payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.395873 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.395844 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume\") pod \"payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.453602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.453565 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgrhd\" (UniqueName: \"kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd\") pod \"payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.472540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.472502 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:33.651719 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:13:33.651678 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83a0c076_8805_4771_8e54_99f1333aff51.slice/crio-971f38799c02043a487c10936c91cd630abefff68e0a15fd88b8e0fd36279463 WatchSource:0}: Error finding container 971f38799c02043a487c10936c91cd630abefff68e0a15fd88b8e0fd36279463: Status 404 returned error can't find the container with id 971f38799c02043a487c10936c91cd630abefff68e0a15fd88b8e0fd36279463 Nov 03 01:13:33.651893 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.651768 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt"] Nov 03 01:13:33.857826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:33.856767 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs"] Nov 03 01:13:33.864461 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:13:33.864423 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76fc554_99de_4249_aed2_5e9d99fe9636.slice/crio-695ff989971c0724c06c2eab6a7afcb8c349113f115d487cc167be6a131c8772 WatchSource:0}: Error finding container 695ff989971c0724c06c2eab6a7afcb8c349113f115d487cc167be6a131c8772: Status 404 returned error can't find the container with id 695ff989971c0724c06c2eab6a7afcb8c349113f115d487cc167be6a131c8772 Nov 03 01:13:34.441800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.441756 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" event={"ID":"83a0c076-8805-4771-8e54-99f1333aff51","Type":"ContainerStarted","Data":"557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e"} Nov 03 01:13:34.442295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.441827 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" event={"ID":"83a0c076-8805-4771-8e54-99f1333aff51","Type":"ContainerStarted","Data":"971f38799c02043a487c10936c91cd630abefff68e0a15fd88b8e0fd36279463"} Nov 03 01:13:34.442295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.441936 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:34.443644 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.443603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" event={"ID":"c76fc554-99de-4249-aed2-5e9d99fe9636","Type":"ContainerStarted","Data":"76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3"} Nov 03 01:13:34.443763 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.443670 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" event={"ID":"c76fc554-99de-4249-aed2-5e9d99fe9636","Type":"ContainerStarted","Data":"695ff989971c0724c06c2eab6a7afcb8c349113f115d487cc167be6a131c8772"} Nov 03 01:13:34.443763 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.443697 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:13:34.559420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.559364 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" podStartSLOduration=2.55934729 podStartE2EDuration="2.55934729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:34.555141097 +0000 UTC m=+4104.700835554" watchObservedRunningTime="2026-11-03 01:13:34.55934729 +0000 UTC m=+4104.705041745" Nov 03 01:13:34.677897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:34.676965 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" podStartSLOduration=1.6768241320000001 podStartE2EDuration="1.676824132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:34.671108861 +0000 UTC m=+4104.816803360" watchObservedRunningTime="2026-11-03 01:13:34.676824132 +0000 UTC m=+4104.822518589" Nov 03 01:13:42.157497 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:13:42.157457 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:13:45.453815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:45.453784 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:13:45.454268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:13:45.453865 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:14:12.155649 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:14:12.155593 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:14:25.055013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:14:25.054974 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw"] Nov 03 01:14:25.102016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:14:25.101979 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e076d9d-bdldw"] Nov 03 01:14:25.999222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:14:25.999177 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569876da-e1c8-48d5-83fb-116a19a4af1a" path="/var/lib/kubelet/pods/569876da-e1c8-48d5-83fb-116a19a4af1a/volumes" Nov 03 01:14:42.158041 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:14:42.157991 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:14:58.419865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:14:58.419812 5144 patch_prober.go:29] interesting pod/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:14:58.420431 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:14:58.419905 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:15:00.179380 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.179343 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm"] Nov 03 01:15:00.182821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.182802 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.188293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.188268 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw"] Nov 03 01:15:00.192289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.192271 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.193243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.193224 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp"] Nov 03 01:15:00.199810 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.196428 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.230985 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.230958 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz"] Nov 03 01:15:00.235260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.235237 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.237775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.237752 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc"] Nov 03 01:15:00.240858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.240838 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.241399 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.241381 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv"] Nov 03 01:15:00.257701 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.255799 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs"] Nov 03 01:15:00.261407 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.258805 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.265990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.265966 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8"] Nov 03 01:15:00.266126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.266084 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.269753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.269732 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.274663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.274643 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:15:00.278712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.278686 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc"] Nov 03 01:15:00.281159 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.281135 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.281274 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.281259 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kh7n\" (UniqueName: \"kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.281326 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.281315 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ktv8\" (UniqueName: \"kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8\") pod \"maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.281374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.281345 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.281447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.281435 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.281497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.281486 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl449\" (UniqueName: \"kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.282676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.282623 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq"] Nov 03 01:15:00.282793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.282710 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.286669 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.286648 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm"] Nov 03 01:15:00.286772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.286718 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.289193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.289171 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw"] Nov 03 01:15:00.317807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.317780 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc"] Nov 03 01:15:00.325398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.325367 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv"] Nov 03 01:15:00.332097 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.332074 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs"] Nov 03 01:15:00.339378 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.339351 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8"] Nov 03 01:15:00.370719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.370694 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz"] Nov 03 01:15:00.382726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.382694 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.382885 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.382780 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.382964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.382881 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-ffxs8\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.383032 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.382967 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl449\" (UniqueName: \"kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.383092 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383012 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tpzb\" (UniqueName: \"kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.383152 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383122 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvwwg\" (UniqueName: \"kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.383216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383181 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.383216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383212 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.383329 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383274 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t579v\" (UniqueName: \"kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.383424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383402 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.383491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383362 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4qb9\" (UniqueName: \"kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.383491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383453 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.383603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383495 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.383603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383512 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.383603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383530 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.383603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383553 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.383603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383582 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kh7n\" (UniqueName: \"kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.383926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383613 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8w8z\" (UniqueName: \"kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z\") pod \"maas-api-key-cleanup-29894475-ffxs8\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.383926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383653 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j89xc\" (UniqueName: \"kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.383926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383655 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.383926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383670 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z42ff\" (UniqueName: \"kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.383926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.383789 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ktv8\" (UniqueName: \"kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8\") pod \"maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.384215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.384042 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.395994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.395967 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp"] Nov 03 01:15:00.402403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.402345 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq"] Nov 03 01:15:00.405587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.405554 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc"] Nov 03 01:15:00.440507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.440474 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl449\" (UniqueName: \"kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449\") pod \"maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.441085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.441058 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ktv8\" (UniqueName: \"kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8\") pod \"maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.441224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.441122 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kh7n\" (UniqueName: \"kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.485061 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485029 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-ffxs8\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.485222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485091 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tpzb\" (UniqueName: \"kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.485222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485146 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvwwg\" (UniqueName: \"kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.485222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485195 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.485382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485250 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t579v\" (UniqueName: \"kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.485382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485277 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4qb9\" (UniqueName: \"kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.485382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485305 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.485382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485350 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.485382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485374 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.485772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485402 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.485772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485439 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.485772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485488 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8w8z\" (UniqueName: \"kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z\") pod \"maas-api-key-cleanup-29894475-ffxs8\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.485772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485511 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j89xc\" (UniqueName: \"kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.485772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485538 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z42ff\" (UniqueName: \"kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.486080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.485936 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894475-ffxs8\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.486080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.486066 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.486234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.486211 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.486302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.486243 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.486356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.486302 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.486512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.486494 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.486661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.486622 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.496578 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.496558 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:00.549990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.549944 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j89xc\" (UniqueName: \"kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc\") pod \"maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.550586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.550547 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tpzb\" (UniqueName: \"kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.553722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.553394 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvwwg\" (UniqueName: \"kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg\") pod \"maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.554189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.554160 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z42ff\" (UniqueName: \"kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.554350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.554323 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t579v\" (UniqueName: \"kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.554671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.554612 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8w8z\" (UniqueName: \"kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z\") pod \"maas-api-key-cleanup-29894475-ffxs8\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.555289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.555269 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4qb9\" (UniqueName: \"kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.574284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.574013 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:00.581689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.581578 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:00.590883 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.590852 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:00.600695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.600387 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:00.607944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.607911 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:00.621651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.621605 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:00.642052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.641847 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:00.658224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.657558 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:00.670100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.667907 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:00.940882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:00.940856 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm"] Nov 03 01:15:01.151962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.151924 5144 generic.go:356] "Generic (PLEG): container finished" podID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerID="547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3" exitCode=0 Nov 03 01:15:01.152125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.152012 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" event={"ID":"d8b57094-a4ae-470d-8842-197f9bd4e7c6","Type":"ContainerDied","Data":"547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3"} Nov 03 01:15:01.152513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.152491 5144 scope.go:122] "RemoveContainer" containerID="547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3" Nov 03 01:15:01.154725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.154690 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:15:01.154836 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.154754 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" event={"ID":"7226c1f9-036f-48c1-a2ac-067f7a436331","Type":"ContainerStarted","Data":"a34f64eec342e4f7862bf37750ad044c9969bd4f88b073a580cae1735afe9029"} Nov 03 01:15:01.154836 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.154770 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:15:01.154836 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.154819 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:15:01.221033 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:15:01.221001 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e7f76d_a799_4120_99a7_f19299a11e72.slice/crio-1ef48285bd726f9ea82bb74d7f64c4ab575ed43f0366814cd34dc19fedee1111 WatchSource:0}: Error finding container 1ef48285bd726f9ea82bb74d7f64c4ab575ed43f0366814cd34dc19fedee1111: Status 404 returned error can't find the container with id 1ef48285bd726f9ea82bb74d7f64c4ab575ed43f0366814cd34dc19fedee1111 Nov 03 01:15:01.222621 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.222578 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp"] Nov 03 01:15:01.234577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.234551 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw"] Nov 03 01:15:01.240470 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:15:01.240446 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db75669_cf47_4fa3_b67f_17f4e4a7c70a.slice/crio-e70bc30d815752b5157572cbcc07452ba2f70f2e1a779f2a83a592346ce0350a WatchSource:0}: Error finding container e70bc30d815752b5157572cbcc07452ba2f70f2e1a779f2a83a592346ce0350a: Status 404 returned error can't find the container with id e70bc30d815752b5157572cbcc07452ba2f70f2e1a779f2a83a592346ce0350a Nov 03 01:15:01.305007 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.298213 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz"] Nov 03 01:15:01.320485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.320450 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc"] Nov 03 01:15:01.329233 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:15:01.329204 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da6cba9_2cc3_4503_9ef5_73ec4b285571.slice/crio-687a0dbc0246d2e266702330d870418650dd20f917b8e894edd4b95ab9f8e063 WatchSource:0}: Error finding container 687a0dbc0246d2e266702330d870418650dd20f917b8e894edd4b95ab9f8e063: Status 404 returned error can't find the container with id 687a0dbc0246d2e266702330d870418650dd20f917b8e894edd4b95ab9f8e063 Nov 03 01:15:01.364299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.364270 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8"] Nov 03 01:15:01.364513 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:15:01.364368 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e39f7c_8d20_4a0a_b6ab_6bdd0ae03fcd.slice/crio-f9fca84f4cd61e1ff592b62af1c7b264198fc72f4dbc442d0a754718826dc590 WatchSource:0}: Error finding container f9fca84f4cd61e1ff592b62af1c7b264198fc72f4dbc442d0a754718826dc590: Status 404 returned error can't find the container with id f9fca84f4cd61e1ff592b62af1c7b264198fc72f4dbc442d0a754718826dc590 Nov 03 01:15:01.418687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.418658 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc"] Nov 03 01:15:01.430247 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.430223 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs"] Nov 03 01:15:01.435714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.435688 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv"] Nov 03 01:15:01.440280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:01.440255 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq"] Nov 03 01:15:01.630930 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:15:01.630899 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bf46411_92e3_4ed3_a679_970f2ec4107e.slice/crio-62530bc52360c697b4fd11347821757df189549b79e76aaf3a3d430f29ef3a6d WatchSource:0}: Error finding container 62530bc52360c697b4fd11347821757df189549b79e76aaf3a3d430f29ef3a6d: Status 404 returned error can't find the container with id 62530bc52360c697b4fd11347821757df189549b79e76aaf3a3d430f29ef3a6d Nov 03 01:15:02.165729 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.165690 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerStarted","Data":"288b18d45bf3f9e29fab7ebcd617105a4279ea31598fe97ec00ca943b75c9490"} Nov 03 01:15:02.165913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.165757 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerStarted","Data":"f9fca84f4cd61e1ff592b62af1c7b264198fc72f4dbc442d0a754718826dc590"} Nov 03 01:15:02.167389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.167362 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" event={"ID":"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09","Type":"ContainerStarted","Data":"e00e0aec52f10693ff7498fbd3982c3c80964f4fa77dd364e75c2a21fee80414"} Nov 03 01:15:02.168668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.168644 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerStarted","Data":"c0f6c273508b90e5486f1cccd309ecb505c2077c9b0ae095d0917c1f74e1bfeb"} Nov 03 01:15:02.168756 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.168690 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerStarted","Data":"687a0dbc0246d2e266702330d870418650dd20f917b8e894edd4b95ab9f8e063"} Nov 03 01:15:02.170575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.170550 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerStarted","Data":"eaee788b15d3803b0a7e2f2f75128ecd7535a7ee63f7b5618a2548fad99e7918"} Nov 03 01:15:02.170679 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.170594 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerStarted","Data":"f9c07bfda7a019538a535ad8721a172a7e61bbaf562a4e1d1f367a6f637638b4"} Nov 03 01:15:02.173125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.173107 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" event={"ID":"d8b57094-a4ae-470d-8842-197f9bd4e7c6","Type":"ContainerStarted","Data":"f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899"} Nov 03 01:15:02.175343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.175317 5144 generic.go:356] "Generic (PLEG): container finished" podID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerID="02c8b5422505cfe89df57341e47654687f8262a124220d074b817eda7dcb585b" exitCode=60 Nov 03 01:15:02.175472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.175444 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" event={"ID":"7226c1f9-036f-48c1-a2ac-067f7a436331","Type":"ContainerDied","Data":"02c8b5422505cfe89df57341e47654687f8262a124220d074b817eda7dcb585b"} Nov 03 01:15:02.175612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.175541 5144 scope.go:122] "RemoveContainer" containerID="02c8b5422505cfe89df57341e47654687f8262a124220d074b817eda7dcb585b" Nov 03 01:15:02.177204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.177179 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerStarted","Data":"36e5e1fdaf237411653af6fac59ebe2c4b1663673d40c17fc1053b4cef909209"} Nov 03 01:15:02.177296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.177224 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerStarted","Data":"1ef48285bd726f9ea82bb74d7f64c4ab575ed43f0366814cd34dc19fedee1111"} Nov 03 01:15:02.178924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.178893 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerStarted","Data":"d539cac89b15f4d197324c887d62c53d854a1f229631189c7e10288df877c547"} Nov 03 01:15:02.179007 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.178936 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerStarted","Data":"e70bc30d815752b5157572cbcc07452ba2f70f2e1a779f2a83a592346ce0350a"} Nov 03 01:15:02.180026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.180007 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" event={"ID":"6bf46411-92e3-4ed3-a679-970f2ec4107e","Type":"ContainerStarted","Data":"62530bc52360c697b4fd11347821757df189549b79e76aaf3a3d430f29ef3a6d"} Nov 03 01:15:02.181136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.181112 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerStarted","Data":"b9c1cf234405aa96d8e5670ab983f609c74d9ea2bac62e52b54ecd87d2b88aa2"} Nov 03 01:15:02.182999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.182979 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerStarted","Data":"a4df0e335b5dec87c52e73e1944fc8971396cbc66295bcfe317c86029203e87e"} Nov 03 01:15:02.308515 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.308455 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" podStartSLOduration=2.308441578 podStartE2EDuration="2.308441578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.307503162 +0000 UTC m=+4192.453197638" watchObservedRunningTime="2026-11-03 01:15:02.308441578 +0000 UTC m=+4192.454136035" Nov 03 01:15:02.320810 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.320720 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:15:02.349669 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.348863 5144 patch_prober.go:29] interesting pod/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 01:15:02.349669 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.348941 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 01:15:02.433183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.420390 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" podStartSLOduration=2.420370171 podStartE2EDuration="2.420370171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.419195942 +0000 UTC m=+4192.564890398" watchObservedRunningTime="2026-11-03 01:15:02.420370171 +0000 UTC m=+4192.566064629" Nov 03 01:15:02.662298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.662251 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" podStartSLOduration=2.662234619 podStartE2EDuration="2.662234619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.544998427 +0000 UTC m=+4192.690692905" watchObservedRunningTime="2026-11-03 01:15:02.662234619 +0000 UTC m=+4192.807929086" Nov 03 01:15:02.906536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:02.906432 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" podStartSLOduration=2.906416765 podStartE2EDuration="2.906416765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.903879488 +0000 UTC m=+4193.049573966" watchObservedRunningTime="2026-11-03 01:15:02.906416765 +0000 UTC m=+4193.052111220" Nov 03 01:15:03.047065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.047009 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" podStartSLOduration=3.046994107 podStartE2EDuration="3.046994107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:03.043108389 +0000 UTC m=+4193.188802862" watchObservedRunningTime="2026-11-03 01:15:03.046994107 +0000 UTC m=+4193.192688563" Nov 03 01:15:03.193460 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.193374 5144 generic.go:356] "Generic (PLEG): container finished" podID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerID="288b18d45bf3f9e29fab7ebcd617105a4279ea31598fe97ec00ca943b75c9490" exitCode=60 Nov 03 01:15:03.193647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.193450 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerDied","Data":"288b18d45bf3f9e29fab7ebcd617105a4279ea31598fe97ec00ca943b75c9490"} Nov 03 01:15:03.193768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.193752 5144 scope.go:122] "RemoveContainer" containerID="288b18d45bf3f9e29fab7ebcd617105a4279ea31598fe97ec00ca943b75c9490" Nov 03 01:15:03.195216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.195193 5144 generic.go:356] "Generic (PLEG): container finished" podID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerID="a9fb83830fe1d931899ee9893fc12d4d2cbd41aa6c1a0ab0e1e8a2b54a8ac9bf" exitCode=60 Nov 03 01:15:03.195340 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.195280 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" event={"ID":"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09","Type":"ContainerDied","Data":"a9fb83830fe1d931899ee9893fc12d4d2cbd41aa6c1a0ab0e1e8a2b54a8ac9bf"} Nov 03 01:15:03.195524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.195500 5144 scope.go:122] "RemoveContainer" containerID="a9fb83830fe1d931899ee9893fc12d4d2cbd41aa6c1a0ab0e1e8a2b54a8ac9bf" Nov 03 01:15:03.197267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.197242 5144 generic.go:356] "Generic (PLEG): container finished" podID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerID="c0f6c273508b90e5486f1cccd309ecb505c2077c9b0ae095d0917c1f74e1bfeb" exitCode=60 Nov 03 01:15:03.197373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.197277 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerDied","Data":"c0f6c273508b90e5486f1cccd309ecb505c2077c9b0ae095d0917c1f74e1bfeb"} Nov 03 01:15:03.197659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.197569 5144 scope.go:122] "RemoveContainer" containerID="c0f6c273508b90e5486f1cccd309ecb505c2077c9b0ae095d0917c1f74e1bfeb" Nov 03 01:15:03.199104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.199085 5144 generic.go:356] "Generic (PLEG): container finished" podID="b30373b8-efc1-491d-b87a-40508b78d75a" containerID="eaee788b15d3803b0a7e2f2f75128ecd7535a7ee63f7b5618a2548fad99e7918" exitCode=60 Nov 03 01:15:03.199198 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.199165 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerDied","Data":"eaee788b15d3803b0a7e2f2f75128ecd7535a7ee63f7b5618a2548fad99e7918"} Nov 03 01:15:03.199359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.199346 5144 scope.go:122] "RemoveContainer" containerID="eaee788b15d3803b0a7e2f2f75128ecd7535a7ee63f7b5618a2548fad99e7918" Nov 03 01:15:03.201179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.201156 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" event={"ID":"7226c1f9-036f-48c1-a2ac-067f7a436331","Type":"ContainerStarted","Data":"98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a"} Nov 03 01:15:03.202903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.202884 5144 generic.go:356] "Generic (PLEG): container finished" podID="68e7f76d-a799-4120-99a7-f19299a11e72" containerID="36e5e1fdaf237411653af6fac59ebe2c4b1663673d40c17fc1053b4cef909209" exitCode=60 Nov 03 01:15:03.202993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.202964 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerDied","Data":"36e5e1fdaf237411653af6fac59ebe2c4b1663673d40c17fc1053b4cef909209"} Nov 03 01:15:03.203177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.203160 5144 scope.go:122] "RemoveContainer" containerID="36e5e1fdaf237411653af6fac59ebe2c4b1663673d40c17fc1053b4cef909209" Nov 03 01:15:03.204601 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.204581 5144 generic.go:356] "Generic (PLEG): container finished" podID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerID="d539cac89b15f4d197324c887d62c53d854a1f229631189c7e10288df877c547" exitCode=60 Nov 03 01:15:03.204719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.204662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerDied","Data":"d539cac89b15f4d197324c887d62c53d854a1f229631189c7e10288df877c547"} Nov 03 01:15:03.204851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.204838 5144 scope.go:122] "RemoveContainer" containerID="d539cac89b15f4d197324c887d62c53d854a1f229631189c7e10288df877c547" Nov 03 01:15:03.207019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.207000 5144 generic.go:356] "Generic (PLEG): container finished" podID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerID="1ca8dc95ef3429bbf258ae617d5edc773425a959d2ddc8cf8d81e6dfcbe1302b" exitCode=60 Nov 03 01:15:03.207114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.207025 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" event={"ID":"6bf46411-92e3-4ed3-a679-970f2ec4107e","Type":"ContainerDied","Data":"1ca8dc95ef3429bbf258ae617d5edc773425a959d2ddc8cf8d81e6dfcbe1302b"} Nov 03 01:15:03.207269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.207254 5144 scope.go:122] "RemoveContainer" containerID="1ca8dc95ef3429bbf258ae617d5edc773425a959d2ddc8cf8d81e6dfcbe1302b" Nov 03 01:15:03.208778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.208758 5144 generic.go:356] "Generic (PLEG): container finished" podID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerID="899b0dbf4a52371036cf532f0eb56221431647db786ca439c700aaaff290fc10" exitCode=60 Nov 03 01:15:03.208864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.208850 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerDied","Data":"899b0dbf4a52371036cf532f0eb56221431647db786ca439c700aaaff290fc10"} Nov 03 01:15:03.209003 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.208986 5144 scope.go:122] "RemoveContainer" containerID="899b0dbf4a52371036cf532f0eb56221431647db786ca439c700aaaff290fc10" Nov 03 01:15:03.210615 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.210596 5144 generic.go:356] "Generic (PLEG): container finished" podID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerID="c81c80f0a086c08ee9d9dd988d65dd3362b076ec170a373eda3f5711a5e84a52" exitCode=60 Nov 03 01:15:03.210727 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.210676 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerDied","Data":"c81c80f0a086c08ee9d9dd988d65dd3362b076ec170a373eda3f5711a5e84a52"} Nov 03 01:15:03.210868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.210852 5144 scope.go:122] "RemoveContainer" containerID="c81c80f0a086c08ee9d9dd988d65dd3362b076ec170a373eda3f5711a5e84a52" Nov 03 01:15:03.351019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.350964 5144 patch_prober.go:29] interesting pod/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 01:15:03.351468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.351045 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 01:15:03.883342 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:03.883285 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" podStartSLOduration=3.883179894 podStartE2EDuration="3.883179894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:03.880304112 +0000 UTC m=+4194.025998571" watchObservedRunningTime="2026-11-03 01:15:03.883179894 +0000 UTC m=+4194.028874354" Nov 03 01:15:04.249659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.245869 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerStarted","Data":"2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf"} Nov 03 01:15:04.250909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.250870 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" event={"ID":"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09","Type":"ContainerStarted","Data":"f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67"} Nov 03 01:15:04.253214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.253146 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerStarted","Data":"0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc"} Nov 03 01:15:04.255405 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.255384 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerStarted","Data":"f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25"} Nov 03 01:15:04.257318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.257297 5144 generic.go:356] "Generic (PLEG): container finished" podID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerID="98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a" exitCode=60 Nov 03 01:15:04.257425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.257376 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" event={"ID":"7226c1f9-036f-48c1-a2ac-067f7a436331","Type":"ContainerDied","Data":"98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a"} Nov 03 01:15:04.257499 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.257444 5144 scope.go:122] "RemoveContainer" containerID="02c8b5422505cfe89df57341e47654687f8262a124220d074b817eda7dcb585b" Nov 03 01:15:04.257610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.257593 5144 scope.go:122] "RemoveContainer" containerID="98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a" Nov 03 01:15:04.257914 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:04.257893 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm_odh-ai-gateway-infra(7226c1f9-036f-48c1-a2ac-067f7a436331)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" Nov 03 01:15:04.259264 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.259244 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerStarted","Data":"69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086"} Nov 03 01:15:04.261465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.261442 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerStarted","Data":"f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76"} Nov 03 01:15:04.263490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.263462 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" event={"ID":"6bf46411-92e3-4ed3-a679-970f2ec4107e","Type":"ContainerStarted","Data":"b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890"} Nov 03 01:15:04.265908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.265882 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerStarted","Data":"1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b"} Nov 03 01:15:04.267533 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.267515 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerStarted","Data":"ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782"} Nov 03 01:15:04.313169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.313128 5144 patch_prober.go:29] interesting pod/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 01:15:04.313313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.313178 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 01:15:04.693453 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:04.693400 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" podStartSLOduration=4.693385086 podStartE2EDuration="4.693385086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:04.691238555 +0000 UTC m=+4194.836933029" watchObservedRunningTime="2026-11-03 01:15:04.693385086 +0000 UTC m=+4194.839079541" Nov 03 01:15:05.049018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.048892 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" podStartSLOduration=5.048871603 podStartE2EDuration="5.048871603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:04.942927851 +0000 UTC m=+4195.088622308" watchObservedRunningTime="2026-11-03 01:15:05.048871603 +0000 UTC m=+4195.194566061" Nov 03 01:15:05.051730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.051701 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd"] Nov 03 01:15:05.058272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.058239 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b5cda05-skjzd"] Nov 03 01:15:05.291609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.291575 5144 generic.go:356] "Generic (PLEG): container finished" podID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerID="b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890" exitCode=60 Nov 03 01:15:05.291831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.291675 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" event={"ID":"6bf46411-92e3-4ed3-a679-970f2ec4107e","Type":"ContainerDied","Data":"b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890"} Nov 03 01:15:05.291831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.291566 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" podStartSLOduration=5.291552747 podStartE2EDuration="5.291552747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:05.290141544 +0000 UTC m=+4195.435836000" watchObservedRunningTime="2026-11-03 01:15:05.291552747 +0000 UTC m=+4195.437247221" Nov 03 01:15:05.291831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.291764 5144 scope.go:122] "RemoveContainer" containerID="1ca8dc95ef3429bbf258ae617d5edc773425a959d2ddc8cf8d81e6dfcbe1302b" Nov 03 01:15:05.292051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.291940 5144 scope.go:122] "RemoveContainer" containerID="b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890" Nov 03 01:15:05.292340 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.292304 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv_odh-ai-gateway-infra(6bf46411-92e3-4ed3-a679-970f2ec4107e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" Nov 03 01:15:05.294743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.294720 5144 generic.go:356] "Generic (PLEG): container finished" podID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerID="1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b" exitCode=60 Nov 03 01:15:05.294862 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.294795 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerDied","Data":"1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b"} Nov 03 01:15:05.295934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.295111 5144 scope.go:122] "RemoveContainer" containerID="1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b" Nov 03 01:15:05.295934 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.295424 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq_odh-ai-gateway-infra(4edbcba3-16f6-4d8f-8589-ef9dd920dfc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" Nov 03 01:15:05.297530 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.297508 5144 generic.go:356] "Generic (PLEG): container finished" podID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerID="ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782" exitCode=60 Nov 03 01:15:05.297676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.297584 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerDied","Data":"ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782"} Nov 03 01:15:05.297947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.297929 5144 scope.go:122] "RemoveContainer" containerID="ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782" Nov 03 01:15:05.298255 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.298236 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs_odh-ai-gateway-infra(276cb1a0-aed3-4155-b71a-0c246b4ff76a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" Nov 03 01:15:05.300142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.300092 5144 generic.go:356] "Generic (PLEG): container finished" podID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerID="2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf" exitCode=60 Nov 03 01:15:05.300299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.300208 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerDied","Data":"2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf"} Nov 03 01:15:05.300688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.300670 5144 scope.go:122] "RemoveContainer" containerID="2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf" Nov 03 01:15:05.301050 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.301030 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-ffxs8_odh-ai-gateway-infra(38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" Nov 03 01:15:05.301759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.301740 5144 generic.go:356] "Generic (PLEG): container finished" podID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerID="f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67" exitCode=60 Nov 03 01:15:05.301853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.301792 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" event={"ID":"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09","Type":"ContainerDied","Data":"f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67"} Nov 03 01:15:05.302041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.302026 5144 scope.go:122] "RemoveContainer" containerID="f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67" Nov 03 01:15:05.302223 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.302207 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc_odh-ai-gateway-infra(cbe0e82d-342d-431b-b8bc-a4bd93fc2a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" Nov 03 01:15:05.303547 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.303519 5144 generic.go:356] "Generic (PLEG): container finished" podID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerID="0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc" exitCode=60 Nov 03 01:15:05.303656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.303589 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerDied","Data":"0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc"} Nov 03 01:15:05.303874 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.303859 5144 scope.go:122] "RemoveContainer" containerID="0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc" Nov 03 01:15:05.304051 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.304034 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc_odh-ai-gateway-infra(9da6cba9-2cc3-4503-9ef5-73ec4b285571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" Nov 03 01:15:05.305537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.305516 5144 generic.go:356] "Generic (PLEG): container finished" podID="b30373b8-efc1-491d-b87a-40508b78d75a" containerID="f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25" exitCode=60 Nov 03 01:15:05.305613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.305603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerDied","Data":"f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25"} Nov 03 01:15:05.305931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.305912 5144 scope.go:122] "RemoveContainer" containerID="f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25" Nov 03 01:15:05.306173 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.306150 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz_odh-ai-gateway-infra(b30373b8-efc1-491d-b87a-40508b78d75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" Nov 03 01:15:05.307683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.307620 5144 scope.go:122] "RemoveContainer" containerID="98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a" Nov 03 01:15:05.307780 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.307761 5144 scope.go:122] "RemoveContainer" containerID="899b0dbf4a52371036cf532f0eb56221431647db786ca439c700aaaff290fc10" Nov 03 01:15:05.307870 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.307846 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm_odh-ai-gateway-infra(7226c1f9-036f-48c1-a2ac-067f7a436331)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" Nov 03 01:15:05.309351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.309330 5144 generic.go:356] "Generic (PLEG): container finished" podID="68e7f76d-a799-4120-99a7-f19299a11e72" containerID="69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086" exitCode=60 Nov 03 01:15:05.309436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.309395 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerDied","Data":"69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086"} Nov 03 01:15:05.309808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.309793 5144 scope.go:122] "RemoveContainer" containerID="69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086" Nov 03 01:15:05.310135 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.310115 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp_odh-ai-gateway-infra(68e7f76d-a799-4120-99a7-f19299a11e72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" Nov 03 01:15:05.311128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.311110 5144 generic.go:356] "Generic (PLEG): container finished" podID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerID="f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76" exitCode=60 Nov 03 01:15:05.311252 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.311126 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerDied","Data":"f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76"} Nov 03 01:15:05.311359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.311309 5144 scope.go:122] "RemoveContainer" containerID="f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76" Nov 03 01:15:05.311681 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:05.311661 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw_odh-ai-gateway-infra(4db75669-cf47-4fa3-b67f-17f4e4a7c70a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" Nov 03 01:15:05.313624 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.313594 5144 patch_prober.go:29] interesting pod/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 01:15:05.313744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.313658 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 01:15:05.347448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.347422 5144 scope.go:122] "RemoveContainer" containerID="c81c80f0a086c08ee9d9dd988d65dd3362b076ec170a373eda3f5711a5e84a52" Nov 03 01:15:05.360829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.360805 5144 scope.go:122] "RemoveContainer" containerID="288b18d45bf3f9e29fab7ebcd617105a4279ea31598fe97ec00ca943b75c9490" Nov 03 01:15:05.373649 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.373603 5144 scope.go:122] "RemoveContainer" containerID="a9fb83830fe1d931899ee9893fc12d4d2cbd41aa6c1a0ab0e1e8a2b54a8ac9bf" Nov 03 01:15:05.386501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.386478 5144 scope.go:122] "RemoveContainer" containerID="c0f6c273508b90e5486f1cccd309ecb505c2077c9b0ae095d0917c1f74e1bfeb" Nov 03 01:15:05.400569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.400544 5144 scope.go:122] "RemoveContainer" containerID="eaee788b15d3803b0a7e2f2f75128ecd7535a7ee63f7b5618a2548fad99e7918" Nov 03 01:15:05.419866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.414007 5144 scope.go:122] "RemoveContainer" containerID="36e5e1fdaf237411653af6fac59ebe2c4b1663673d40c17fc1053b4cef909209" Nov 03 01:15:05.452256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.452227 5144 scope.go:122] "RemoveContainer" containerID="d539cac89b15f4d197324c887d62c53d854a1f229631189c7e10288df877c547" Nov 03 01:15:05.539658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.532690 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" podStartSLOduration=5.53266902 podStartE2EDuration="5.53266902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:05.513546719 +0000 UTC m=+4195.659241177" watchObservedRunningTime="2026-11-03 01:15:05.53266902 +0000 UTC m=+4195.678363479" Nov 03 01:15:05.999409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:05.999377 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc3ff9e-725c-4eb3-9f25-a88a55f1819a" path="/var/lib/kubelet/pods/ffc3ff9e-725c-4eb3-9f25-a88a55f1819a/volumes" Nov 03 01:15:06.313604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.313571 5144 patch_prober.go:29] interesting pod/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" start-of-body= Nov 03 01:15:06.313869 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.313669 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.63:15021/healthz/ready\": dial tcp 10.132.1.63:15021: connect: connection refused" Nov 03 01:15:06.351197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.351165 5144 scope.go:122] "RemoveContainer" containerID="f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67" Nov 03 01:15:06.351598 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.351573 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc_odh-ai-gateway-infra(cbe0e82d-342d-431b-b8bc-a4bd93fc2a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" Nov 03 01:15:06.353095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.353072 5144 scope.go:122] "RemoveContainer" containerID="0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc" Nov 03 01:15:06.353370 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.353350 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc_odh-ai-gateway-infra(9da6cba9-2cc3-4503-9ef5-73ec4b285571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" Nov 03 01:15:06.355000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.354984 5144 scope.go:122] "RemoveContainer" containerID="f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25" Nov 03 01:15:06.355456 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.355433 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz_odh-ai-gateway-infra(b30373b8-efc1-491d-b87a-40508b78d75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" Nov 03 01:15:06.356246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.356227 5144 scope.go:122] "RemoveContainer" containerID="69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086" Nov 03 01:15:06.356443 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.356426 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp_odh-ai-gateway-infra(68e7f76d-a799-4120-99a7-f19299a11e72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" Nov 03 01:15:06.357968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.357953 5144 scope.go:122] "RemoveContainer" containerID="f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76" Nov 03 01:15:06.358239 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.358218 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw_odh-ai-gateway-infra(4db75669-cf47-4fa3-b67f-17f4e4a7c70a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" Nov 03 01:15:06.359716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.359700 5144 scope.go:122] "RemoveContainer" containerID="b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890" Nov 03 01:15:06.359928 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.359905 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv_odh-ai-gateway-infra(6bf46411-92e3-4ed3-a679-970f2ec4107e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" Nov 03 01:15:06.361193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.361174 5144 scope.go:122] "RemoveContainer" containerID="1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b" Nov 03 01:15:06.361555 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.361531 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq_odh-ai-gateway-infra(4edbcba3-16f6-4d8f-8589-ef9dd920dfc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" Nov 03 01:15:06.362812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.362794 5144 scope.go:122] "RemoveContainer" containerID="ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782" Nov 03 01:15:06.362986 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.362965 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs_odh-ai-gateway-infra(276cb1a0-aed3-4155-b71a-0c246b4ff76a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" Nov 03 01:15:06.364374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:06.364360 5144 scope.go:122] "RemoveContainer" containerID="2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf" Nov 03 01:15:06.364531 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:06.364510 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-ffxs8_odh-ai-gateway-infra(38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" Nov 03 01:15:07.318454 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:07.318415 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:15:07.319003 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:07.318752 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:15:07.319549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:07.319528 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:15:11.065785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:11.065750 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:15:11.069826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:11.069800 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:15:12.155955 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:12.155918 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:15:15.989133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:15.989100 5144 scope.go:122] "RemoveContainer" containerID="98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a" Nov 03 01:15:15.989679 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:15.989416 5144 scope.go:122] "RemoveContainer" containerID="1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b" Nov 03 01:15:15.989679 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:15.989559 5144 scope.go:122] "RemoveContainer" containerID="ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782" Nov 03 01:15:16.444717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:16.444682 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerStarted","Data":"1db3300883a66cf078e8775796eff262a1c3f6edbbc38f1302e3211e1bf01325"} Nov 03 01:15:16.446930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:16.446898 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerStarted","Data":"c7c30eb4af70548f2cb32aa2261333cb7173fa81c90590045a67a88bd847d7ee"} Nov 03 01:15:16.989130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:16.989090 5144 scope.go:122] "RemoveContainer" containerID="69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086" Nov 03 01:15:17.456833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.456781 5144 generic.go:356] "Generic (PLEG): container finished" podID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerID="682caec9a3fa5752d73650d4480fa6ee15d9f2ec25f8b8bd757e9c48a7975310" exitCode=60 Nov 03 01:15:17.457317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.456852 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" event={"ID":"7226c1f9-036f-48c1-a2ac-067f7a436331","Type":"ContainerDied","Data":"682caec9a3fa5752d73650d4480fa6ee15d9f2ec25f8b8bd757e9c48a7975310"} Nov 03 01:15:17.457317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.456918 5144 scope.go:122] "RemoveContainer" containerID="98548355160be55b0c8183e9afc42709867ec94209b88534616b2456ad8bb00a" Nov 03 01:15:17.457472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.457372 5144 scope.go:122] "RemoveContainer" containerID="682caec9a3fa5752d73650d4480fa6ee15d9f2ec25f8b8bd757e9c48a7975310" Nov 03 01:15:17.457858 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:17.457817 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm_odh-ai-gateway-infra(7226c1f9-036f-48c1-a2ac-067f7a436331)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" Nov 03 01:15:17.460540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.460516 5144 generic.go:356] "Generic (PLEG): container finished" podID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerID="1db3300883a66cf078e8775796eff262a1c3f6edbbc38f1302e3211e1bf01325" exitCode=60 Nov 03 01:15:17.460675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.460574 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerDied","Data":"1db3300883a66cf078e8775796eff262a1c3f6edbbc38f1302e3211e1bf01325"} Nov 03 01:15:17.461059 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.461039 5144 scope.go:122] "RemoveContainer" containerID="1db3300883a66cf078e8775796eff262a1c3f6edbbc38f1302e3211e1bf01325" Nov 03 01:15:17.461384 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:17.461363 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq_odh-ai-gateway-infra(4edbcba3-16f6-4d8f-8589-ef9dd920dfc1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" Nov 03 01:15:17.462994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.462969 5144 generic.go:356] "Generic (PLEG): container finished" podID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerID="c7c30eb4af70548f2cb32aa2261333cb7173fa81c90590045a67a88bd847d7ee" exitCode=60 Nov 03 01:15:17.463086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.463038 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerDied","Data":"c7c30eb4af70548f2cb32aa2261333cb7173fa81c90590045a67a88bd847d7ee"} Nov 03 01:15:17.463314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.463301 5144 scope.go:122] "RemoveContainer" containerID="c7c30eb4af70548f2cb32aa2261333cb7173fa81c90590045a67a88bd847d7ee" Nov 03 01:15:17.463539 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:17.463523 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs_odh-ai-gateway-infra(276cb1a0-aed3-4155-b71a-0c246b4ff76a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" Nov 03 01:15:17.471513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.471496 5144 scope.go:122] "RemoveContainer" containerID="1a35bc598a62dbfa7d2c491bd5af2d7b6b0dc0b22a2e9f0d4645fad76b7e6b6b" Nov 03 01:15:17.483020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.482997 5144 scope.go:122] "RemoveContainer" containerID="ba1407641e97ea3a009bf68662ef9dbf7a6c785f549b96c72c62568549f5f782" Nov 03 01:15:17.568840 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.568807 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq"] Nov 03 01:15:17.681348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.681295 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs"] Nov 03 01:15:17.989741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.989702 5144 scope.go:122] "RemoveContainer" containerID="b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890" Nov 03 01:15:17.989920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.989863 5144 scope.go:122] "RemoveContainer" containerID="0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc" Nov 03 01:15:17.990178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.990159 5144 scope.go:122] "RemoveContainer" containerID="f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76" Nov 03 01:15:17.991273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.990625 5144 scope.go:122] "RemoveContainer" containerID="f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67" Nov 03 01:15:17.991273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:17.990707 5144 scope.go:122] "RemoveContainer" containerID="f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25" Nov 03 01:15:18.475578 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.475548 5144 generic.go:356] "Generic (PLEG): container finished" podID="68e7f76d-a799-4120-99a7-f19299a11e72" containerID="f3d5effbf5dace75f967d95a5e78e839e762f6d93c2c3198586676bd4089d64d" exitCode=60 Nov 03 01:15:18.476006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.475624 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerDied","Data":"f3d5effbf5dace75f967d95a5e78e839e762f6d93c2c3198586676bd4089d64d"} Nov 03 01:15:18.476006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.475699 5144 scope.go:122] "RemoveContainer" containerID="69fdcd3b92b3a6be33a0aae0f0c7bc2a66ca2a12d1f3665f6a7eb572fe78f086" Nov 03 01:15:18.476153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.476098 5144 scope.go:122] "RemoveContainer" containerID="f3d5effbf5dace75f967d95a5e78e839e762f6d93c2c3198586676bd4089d64d" Nov 03 01:15:18.476470 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:18.476443 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp_odh-ai-gateway-infra(68e7f76d-a799-4120-99a7-f19299a11e72)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" Nov 03 01:15:18.590209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.590126 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm"] Nov 03 01:15:18.821316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.820822 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:18.832234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.832212 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:18.844500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.844441 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:18.906575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.906548 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8\" (UniqueName: \"kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8\") pod \"7226c1f9-036f-48c1-a2ac-067f7a436331\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " Nov 03 01:15:18.906760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.906678 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca\") pod \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " Nov 03 01:15:18.906760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.906698 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc\" (UniqueName: \"kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc\") pod \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " Nov 03 01:15:18.906842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.906825 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca\") pod \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\" (UID: \"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1\") " Nov 03 01:15:18.906893 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.906852 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca\") pod \"7226c1f9-036f-48c1-a2ac-067f7a436331\" (UID: \"7226c1f9-036f-48c1-a2ac-067f7a436331\") " Nov 03 01:15:18.906948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.906908 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff\" (UniqueName: \"kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff\") pod \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\" (UID: \"276cb1a0-aed3-4155-b71a-0c246b4ff76a\") " Nov 03 01:15:18.907184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.907157 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca" pod "276cb1a0-aed3-4155-b71a-0c246b4ff76a" (UID: "276cb1a0-aed3-4155-b71a-0c246b4ff76a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:18.907320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.907212 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca" pod "4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" (UID: "4edbcba3-16f6-4d8f-8589-ef9dd920dfc1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:18.907320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.907254 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca" pod "7226c1f9-036f-48c1-a2ac-067f7a436331" (UID: "7226c1f9-036f-48c1-a2ac-067f7a436331"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:18.907450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.907437 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/276cb1a0-aed3-4155-b71a-0c246b4ff76a-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:18.907489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.907454 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:18.907489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.907463 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7226c1f9-036f-48c1-a2ac-067f7a436331-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:18.909170 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.909152 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff" pod "276cb1a0-aed3-4155-b71a-0c246b4ff76a" (UID: "276cb1a0-aed3-4155-b71a-0c246b4ff76a"). InnerVolumeSpecName "kube-api-access-z42ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:18.909436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.909418 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8" pod "7226c1f9-036f-48c1-a2ac-067f7a436331" (UID: "7226c1f9-036f-48c1-a2ac-067f7a436331"). InnerVolumeSpecName "kube-api-access-9ktv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:18.909480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:18.909445 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc" pod "4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" (UID: "4edbcba3-16f6-4d8f-8589-ef9dd920dfc1"). InnerVolumeSpecName "kube-api-access-j89xc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.006798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.006760 5144 scope.go:122] "RemoveContainer" containerID="2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf" Nov 03 01:15:19.011239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.011204 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ktv8\" (UniqueName: \"kubernetes.io/projected/7226c1f9-036f-48c1-a2ac-067f7a436331-kube-api-access-9ktv8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.011396 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.011241 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j89xc\" (UniqueName: \"kubernetes.io/projected/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1-kube-api-access-j89xc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.011396 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.011257 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z42ff\" (UniqueName: \"kubernetes.io/projected/276cb1a0-aed3-4155-b71a-0c246b4ff76a-kube-api-access-z42ff\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.492876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.492784 5144 generic.go:356] "Generic (PLEG): container finished" podID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerID="b58690e9e207f6785b725aa5228c47c931960c29609b3ef81817aedfed3b7347" exitCode=60 Nov 03 01:15:19.492876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.492858 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" event={"ID":"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09","Type":"ContainerDied","Data":"b58690e9e207f6785b725aa5228c47c931960c29609b3ef81817aedfed3b7347"} Nov 03 01:15:19.493396 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.492932 5144 scope.go:122] "RemoveContainer" containerID="f2d7e6384153df8701ce95d1491140e54877e5161d7b9e684aa262502594ce67" Nov 03 01:15:19.493396 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.493363 5144 scope.go:122] "RemoveContainer" containerID="b58690e9e207f6785b725aa5228c47c931960c29609b3ef81817aedfed3b7347" Nov 03 01:15:19.493810 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:19.493788 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc_odh-ai-gateway-infra(cbe0e82d-342d-431b-b8bc-a4bd93fc2a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" Nov 03 01:15:19.494999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.494978 5144 generic.go:356] "Generic (PLEG): container finished" podID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerID="cc06072d17888678f1ace1478244a0302abf1315bb61be65cff0f18c197ff21f" exitCode=60 Nov 03 01:15:19.495123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.495043 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerDied","Data":"cc06072d17888678f1ace1478244a0302abf1315bb61be65cff0f18c197ff21f"} Nov 03 01:15:19.495406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.495393 5144 scope.go:122] "RemoveContainer" containerID="cc06072d17888678f1ace1478244a0302abf1315bb61be65cff0f18c197ff21f" Nov 03 01:15:19.495724 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:19.495705 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc_odh-ai-gateway-infra(9da6cba9-2cc3-4503-9ef5-73ec4b285571)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" Nov 03 01:15:19.496897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.496877 5144 generic.go:356] "Generic (PLEG): container finished" podID="b30373b8-efc1-491d-b87a-40508b78d75a" containerID="a3eae4c9fcc1db64017e5efe95e12ad0b1bdf3290bf179ab9c4db811d5b28366" exitCode=60 Nov 03 01:15:19.497020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.496952 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerDied","Data":"a3eae4c9fcc1db64017e5efe95e12ad0b1bdf3290bf179ab9c4db811d5b28366"} Nov 03 01:15:19.497305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.497289 5144 scope.go:122] "RemoveContainer" containerID="a3eae4c9fcc1db64017e5efe95e12ad0b1bdf3290bf179ab9c4db811d5b28366" Nov 03 01:15:19.497586 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:19.497556 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz_odh-ai-gateway-infra(b30373b8-efc1-491d-b87a-40508b78d75a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" Nov 03 01:15:19.498253 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.498233 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" Nov 03 01:15:19.498368 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.498253 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm" event={"ID":"7226c1f9-036f-48c1-a2ac-067f7a436331","Type":"ContainerDied","Data":"a34f64eec342e4f7862bf37750ad044c9969bd4f88b073a580cae1735afe9029"} Nov 03 01:15:19.501541 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.501524 5144 generic.go:356] "Generic (PLEG): container finished" podID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerID="742dc5ec0d3f50b6b5279c71043c6a383f701b83d06dea41830a36521a558160" exitCode=60 Nov 03 01:15:19.501656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.501588 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerDied","Data":"742dc5ec0d3f50b6b5279c71043c6a383f701b83d06dea41830a36521a558160"} Nov 03 01:15:19.501961 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.501938 5144 scope.go:122] "RemoveContainer" containerID="742dc5ec0d3f50b6b5279c71043c6a383f701b83d06dea41830a36521a558160" Nov 03 01:15:19.502196 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:19.502174 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw_odh-ai-gateway-infra(4db75669-cf47-4fa3-b67f-17f4e4a7c70a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" Nov 03 01:15:19.503488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.503462 5144 generic.go:356] "Generic (PLEG): container finished" podID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerID="8e3d3d7c3eb49ea24d935950d12cfe3e047c74d3a0b6d9bf0a5409f4b11794f1" exitCode=60 Nov 03 01:15:19.503570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.503546 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" event={"ID":"6bf46411-92e3-4ed3-a679-970f2ec4107e","Type":"ContainerDied","Data":"8e3d3d7c3eb49ea24d935950d12cfe3e047c74d3a0b6d9bf0a5409f4b11794f1"} Nov 03 01:15:19.503891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.503875 5144 scope.go:122] "RemoveContainer" containerID="8e3d3d7c3eb49ea24d935950d12cfe3e047c74d3a0b6d9bf0a5409f4b11794f1" Nov 03 01:15:19.504124 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:19.504103 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv_odh-ai-gateway-infra(6bf46411-92e3-4ed3-a679-970f2ec4107e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" Nov 03 01:15:19.504946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.504927 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" Nov 03 01:15:19.504946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.504937 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq" event={"ID":"4edbcba3-16f6-4d8f-8589-ef9dd920dfc1","Type":"ContainerDied","Data":"b9c1cf234405aa96d8e5670ab983f609c74d9ea2bac62e52b54ecd87d2b88aa2"} Nov 03 01:15:19.506428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.506407 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" event={"ID":"276cb1a0-aed3-4155-b71a-0c246b4ff76a","Type":"ContainerDied","Data":"a4df0e335b5dec87c52e73e1944fc8971396cbc66295bcfe317c86029203e87e"} Nov 03 01:15:19.506540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.506455 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs" Nov 03 01:15:19.513038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.513022 5144 scope.go:122] "RemoveContainer" containerID="0424825c6803de1c8068a7d3bc9bd4358a2f0d98941afa27a79ac9224733e5fc" Nov 03 01:15:19.529703 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.529685 5144 scope.go:122] "RemoveContainer" containerID="f06f4697069665357061147fde574eac96f6dec087c6aac3cbd6a21f181d0c25" Nov 03 01:15:19.544158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.544137 5144 scope.go:122] "RemoveContainer" containerID="682caec9a3fa5752d73650d4480fa6ee15d9f2ec25f8b8bd757e9c48a7975310" Nov 03 01:15:19.556365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.556338 5144 scope.go:122] "RemoveContainer" containerID="f2387071d15a7ccb719a0eef339bc302189be94b764247ffef7ffaf875476f76" Nov 03 01:15:19.569083 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.569062 5144 scope.go:122] "RemoveContainer" containerID="b42110e35b08a134bb34658049c03b419e81461650ea56f980031dc8f2144890" Nov 03 01:15:19.581382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.581362 5144 scope.go:122] "RemoveContainer" containerID="1db3300883a66cf078e8775796eff262a1c3f6edbbc38f1302e3211e1bf01325" Nov 03 01:15:19.591361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.591341 5144 scope.go:122] "RemoveContainer" containerID="c7c30eb4af70548f2cb32aa2261333cb7173fa81c90590045a67a88bd847d7ee" Nov 03 01:15:19.604901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.604877 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp"] Nov 03 01:15:19.780831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.780806 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:19.840800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.840768 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs"] Nov 03 01:15:19.852385 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.852354 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-20f0a3-29894475-kdmzs"] Nov 03 01:15:19.930768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.930736 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca\") pod \"68e7f76d-a799-4120-99a7-f19299a11e72\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " Nov 03 01:15:19.931021 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.931002 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n\" (UniqueName: \"kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n\") pod \"68e7f76d-a799-4120-99a7-f19299a11e72\" (UID: \"68e7f76d-a799-4120-99a7-f19299a11e72\") " Nov 03 01:15:19.931164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.931140 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca" pod "68e7f76d-a799-4120-99a7-f19299a11e72" (UID: "68e7f76d-a799-4120-99a7-f19299a11e72"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:19.931606 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.931579 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68e7f76d-a799-4120-99a7-f19299a11e72-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:19.933243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.933217 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n" pod "68e7f76d-a799-4120-99a7-f19299a11e72" (UID: "68e7f76d-a799-4120-99a7-f19299a11e72"). InnerVolumeSpecName "kube-api-access-9kh7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:19.998689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:19.998603 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" path="/var/lib/kubelet/pods/276cb1a0-aed3-4155-b71a-0c246b4ff76a/volumes" Nov 03 01:15:20.033117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.033092 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kh7n\" (UniqueName: \"kubernetes.io/projected/68e7f76d-a799-4120-99a7-f19299a11e72-kube-api-access-9kh7n\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:20.070085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.070056 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq"] Nov 03 01:15:20.074000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.073973 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-531ee4-29894475-9t6mq"] Nov 03 01:15:20.182658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.182593 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm"] Nov 03 01:15:20.186210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.186187 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-aee0dd37-29894475-nkrrm"] Nov 03 01:15:20.516019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.515986 5144 generic.go:356] "Generic (PLEG): container finished" podID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerID="77304de155b44b3a5d97007f2c04340abe4ae57f3856cf05710186d4355ef2d5" exitCode=60 Nov 03 01:15:20.516497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.516067 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerDied","Data":"77304de155b44b3a5d97007f2c04340abe4ae57f3856cf05710186d4355ef2d5"} Nov 03 01:15:20.516497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.516139 5144 scope.go:122] "RemoveContainer" containerID="2afc5dc338a1c1db27b4cb81f0d64495754f979a1420d14d0de4e07d4a968dbf" Nov 03 01:15:20.516665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.516564 5144 scope.go:122] "RemoveContainer" containerID="77304de155b44b3a5d97007f2c04340abe4ae57f3856cf05710186d4355ef2d5" Nov 03 01:15:20.516974 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:20.516950 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894475-ffxs8_odh-ai-gateway-infra(38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" Nov 03 01:15:20.531316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.531284 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" Nov 03 01:15:20.531465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.531281 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp" event={"ID":"68e7f76d-a799-4120-99a7-f19299a11e72","Type":"ContainerDied","Data":"1ef48285bd726f9ea82bb74d7f64c4ab575ed43f0366814cd34dc19fedee1111"} Nov 03 01:15:20.531722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.531704 5144 scope.go:122] "RemoveContainer" containerID="f3d5effbf5dace75f967d95a5e78e839e762f6d93c2c3198586676bd4089d64d" Nov 03 01:15:20.617789 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.617753 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc"] Nov 03 01:15:20.745422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.745051 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw"] Nov 03 01:15:20.823970 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.823937 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:20.889262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.889222 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp"] Nov 03 01:15:20.896513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.896483 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-e4c8b1-29894475-j4xhp"] Nov 03 01:15:20.937933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.937907 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:20.951899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.951877 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb\" (UniqueName: \"kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb\") pod \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " Nov 03 01:15:20.952060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.951965 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca\") pod \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\" (UID: \"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09\") " Nov 03 01:15:20.952366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.952335 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca" pod "cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" (UID: "cbe0e82d-342d-431b-b8bc-a4bd93fc2a09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:20.954113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.954094 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb" pod "cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" (UID: "cbe0e82d-342d-431b-b8bc-a4bd93fc2a09"). InnerVolumeSpecName "kube-api-access-9tpzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:20.970825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:20.970800 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc"] Nov 03 01:15:21.055132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.053415 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca\") pod \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " Nov 03 01:15:21.055132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.053517 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449\" (UniqueName: \"kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449\") pod \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\" (UID: \"4db75669-cf47-4fa3-b67f-17f4e4a7c70a\") " Nov 03 01:15:21.055369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.055131 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tpzb\" (UniqueName: \"kubernetes.io/projected/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-kube-api-access-9tpzb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.055369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.055157 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.057311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.057043 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca" pod "4db75669-cf47-4fa3-b67f-17f4e4a7c70a" (UID: "4db75669-cf47-4fa3-b67f-17f4e4a7c70a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:21.068092 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.068061 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449" pod "4db75669-cf47-4fa3-b67f-17f4e4a7c70a" (UID: "4db75669-cf47-4fa3-b67f-17f4e4a7c70a"). InnerVolumeSpecName "kube-api-access-bl449". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:21.153964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.153931 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:21.156266 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.156239 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.156457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.156437 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl449\" (UniqueName: \"kubernetes.io/projected/4db75669-cf47-4fa3-b67f-17f4e4a7c70a-kube-api-access-bl449\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.257687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.257625 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg\" (UniqueName: \"kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg\") pod \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " Nov 03 01:15:21.257961 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.257944 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca\") pod \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\" (UID: \"9da6cba9-2cc3-4503-9ef5-73ec4b285571\") " Nov 03 01:15:21.258438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.258412 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca" pod "9da6cba9-2cc3-4503-9ef5-73ec4b285571" (UID: "9da6cba9-2cc3-4503-9ef5-73ec4b285571"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:21.260602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.260570 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg" pod "9da6cba9-2cc3-4503-9ef5-73ec4b285571" (UID: "9da6cba9-2cc3-4503-9ef5-73ec4b285571"). InnerVolumeSpecName "kube-api-access-fvwwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:21.305133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.305054 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv"] Nov 03 01:15:21.359807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.359766 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9da6cba9-2cc3-4503-9ef5-73ec4b285571-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.359807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.359807 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvwwg\" (UniqueName: \"kubernetes.io/projected/9da6cba9-2cc3-4503-9ef5-73ec4b285571-kube-api-access-fvwwg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.422710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.422676 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz"] Nov 03 01:15:21.488489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.488461 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:21.552003 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.551966 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" event={"ID":"cbe0e82d-342d-431b-b8bc-a4bd93fc2a09","Type":"ContainerDied","Data":"e00e0aec52f10693ff7498fbd3982c3c80964f4fa77dd364e75c2a21fee80414"} Nov 03 01:15:21.552565 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.552538 5144 scope.go:122] "RemoveContainer" containerID="b58690e9e207f6785b725aa5228c47c931960c29609b3ef81817aedfed3b7347" Nov 03 01:15:21.552565 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.552043 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc" Nov 03 01:15:21.561611 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.561541 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9\" (UniqueName: \"kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9\") pod \"6bf46411-92e3-4ed3-a679-970f2ec4107e\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " Nov 03 01:15:21.561749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.561666 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca\") pod \"6bf46411-92e3-4ed3-a679-970f2ec4107e\" (UID: \"6bf46411-92e3-4ed3-a679-970f2ec4107e\") " Nov 03 01:15:21.562215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.562188 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" event={"ID":"9da6cba9-2cc3-4503-9ef5-73ec4b285571","Type":"ContainerDied","Data":"687a0dbc0246d2e266702330d870418650dd20f917b8e894edd4b95ab9f8e063"} Nov 03 01:15:21.562313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.562287 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc" Nov 03 01:15:21.562945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.562917 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca" pod "6bf46411-92e3-4ed3-a679-970f2ec4107e" (UID: "6bf46411-92e3-4ed3-a679-970f2ec4107e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:21.565590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.565562 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9" pod "6bf46411-92e3-4ed3-a679-970f2ec4107e" (UID: "6bf46411-92e3-4ed3-a679-970f2ec4107e"). InnerVolumeSpecName "kube-api-access-d4qb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:21.578951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.578921 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" event={"ID":"4db75669-cf47-4fa3-b67f-17f4e4a7c70a","Type":"ContainerDied","Data":"e70bc30d815752b5157572cbcc07452ba2f70f2e1a779f2a83a592346ce0350a"} Nov 03 01:15:21.579078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.578928 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw" Nov 03 01:15:21.581920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.581830 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" event={"ID":"6bf46411-92e3-4ed3-a679-970f2ec4107e","Type":"ContainerDied","Data":"62530bc52360c697b4fd11347821757df189549b79e76aaf3a3d430f29ef3a6d"} Nov 03 01:15:21.581920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.581913 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv" Nov 03 01:15:21.588499 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.588477 5144 scope.go:122] "RemoveContainer" containerID="cc06072d17888678f1ace1478244a0302abf1315bb61be65cff0f18c197ff21f" Nov 03 01:15:21.600429 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.600410 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:21.603176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.603159 5144 scope.go:122] "RemoveContainer" containerID="742dc5ec0d3f50b6b5279c71043c6a383f701b83d06dea41830a36521a558160" Nov 03 01:15:21.612861 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.612845 5144 scope.go:122] "RemoveContainer" containerID="8e3d3d7c3eb49ea24d935950d12cfe3e047c74d3a0b6d9bf0a5409f4b11794f1" Nov 03 01:15:21.655853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.655826 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8"] Nov 03 01:15:21.662955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.662929 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca\") pod \"b30373b8-efc1-491d-b87a-40508b78d75a\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " Nov 03 01:15:21.663064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.663048 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v\" (UniqueName: \"kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v\") pod \"b30373b8-efc1-491d-b87a-40508b78d75a\" (UID: \"b30373b8-efc1-491d-b87a-40508b78d75a\") " Nov 03 01:15:21.663315 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.663290 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca" pod "b30373b8-efc1-491d-b87a-40508b78d75a" (UID: "b30373b8-efc1-491d-b87a-40508b78d75a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:21.663578 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.663557 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4qb9\" (UniqueName: \"kubernetes.io/projected/6bf46411-92e3-4ed3-a679-970f2ec4107e-kube-api-access-d4qb9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.663736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.663580 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bf46411-92e3-4ed3-a679-970f2ec4107e-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.663736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.663594 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b30373b8-efc1-491d-b87a-40508b78d75a-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.665689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.665617 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v" pod "b30373b8-efc1-491d-b87a-40508b78d75a" (UID: "b30373b8-efc1-491d-b87a-40508b78d75a"). InnerVolumeSpecName "kube-api-access-t579v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:21.770695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.768842 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t579v\" (UniqueName: \"kubernetes.io/projected/b30373b8-efc1-491d-b87a-40508b78d75a-kube-api-access-t579v\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.802427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.802401 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:21.869878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.869794 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z\" (UniqueName: \"kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z\") pod \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " Nov 03 01:15:21.870054 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.870002 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca\") pod \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\" (UID: \"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd\") " Nov 03 01:15:21.870307 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.870283 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca" pod "38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" (UID: "38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:21.870529 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.870514 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:21.872053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.872030 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z" pod "38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" (UID: "38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd"). InnerVolumeSpecName "kube-api-access-s8w8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:21.953041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.952760 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc"] Nov 03 01:15:21.973741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:21.973245 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8w8z\" (UniqueName: \"kubernetes.io/projected/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd-kube-api-access-s8w8z\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:22.006655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.004942 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" path="/var/lib/kubelet/pods/4edbcba3-16f6-4d8f-8589-ef9dd920dfc1/volumes" Nov 03 01:15:22.006655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.005600 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" path="/var/lib/kubelet/pods/68e7f76d-a799-4120-99a7-f19299a11e72/volumes" Nov 03 01:15:22.024656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.022501 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" path="/var/lib/kubelet/pods/7226c1f9-036f-48c1-a2ac-067f7a436331/volumes" Nov 03 01:15:22.024656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.023518 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-328e1b-29894475-gbqwc"] Nov 03 01:15:22.082747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.079690 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw"] Nov 03 01:15:22.082747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.082581 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-e28e1a-29894475-l7gtw"] Nov 03 01:15:22.179366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.179237 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc"] Nov 03 01:15:22.187623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.187590 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-ebeeb9-29894475-w47qc"] Nov 03 01:15:22.472758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.472666 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv"] Nov 03 01:15:22.476291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.476261 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-fce94b-29894475-5n7cv"] Nov 03 01:15:22.597198 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.595787 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" Nov 03 01:15:22.597198 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.595815 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz" event={"ID":"b30373b8-efc1-491d-b87a-40508b78d75a","Type":"ContainerDied","Data":"f9c07bfda7a019538a535ad8721a172a7e61bbaf562a4e1d1f367a6f637638b4"} Nov 03 01:15:22.597198 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.595875 5144 scope.go:122] "RemoveContainer" containerID="a3eae4c9fcc1db64017e5efe95e12ad0b1bdf3290bf179ab9c4db811d5b28366" Nov 03 01:15:22.601225 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.601100 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" event={"ID":"38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd","Type":"ContainerDied","Data":"f9fca84f4cd61e1ff592b62af1c7b264198fc72f4dbc442d0a754718826dc590"} Nov 03 01:15:22.601225 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.601132 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8" Nov 03 01:15:22.629149 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.609128 5144 scope.go:122] "RemoveContainer" containerID="77304de155b44b3a5d97007f2c04340abe4ae57f3856cf05710186d4355ef2d5" Nov 03 01:15:22.764048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.764013 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8"] Nov 03 01:15:22.770644 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.770595 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894475-ffxs8"] Nov 03 01:15:22.885735 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.878120 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz"] Nov 03 01:15:22.885735 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:22.884291 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-b2e795-29894475-xv5qz"] Nov 03 01:15:23.998180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:23.998143 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" path="/var/lib/kubelet/pods/38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd/volumes" Nov 03 01:15:23.998604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:23.998589 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" path="/var/lib/kubelet/pods/4db75669-cf47-4fa3-b67f-17f4e4a7c70a/volumes" Nov 03 01:15:23.998977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:23.998965 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" path="/var/lib/kubelet/pods/6bf46411-92e3-4ed3-a679-970f2ec4107e/volumes" Nov 03 01:15:23.999609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:23.999594 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" path="/var/lib/kubelet/pods/9da6cba9-2cc3-4503-9ef5-73ec4b285571/volumes" Nov 03 01:15:24.000005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:23.999990 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" path="/var/lib/kubelet/pods/b30373b8-efc1-491d-b87a-40508b78d75a/volumes" Nov 03 01:15:24.000346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:24.000334 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" path="/var/lib/kubelet/pods/cbe0e82d-342d-431b-b8bc-a4bd93fc2a09/volumes" Nov 03 01:15:42.158420 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:15:42.158378 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:15:49.136956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.136921 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb"] Nov 03 01:15:49.137830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137807 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.137830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137830 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137852 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137859 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137872 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137879 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137888 5144 cpu_manager.go:414] "containerMap: removing container" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137894 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137909 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137915 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137932 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137939 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137954 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137960 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137971 5144 cpu_manager.go:414] "containerMap: removing container" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.137978 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138000 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138006 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138017 5144 cpu_manager.go:414] "containerMap: removing container" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.138025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138024 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138035 5144 cpu_manager.go:414] "containerMap: removing container" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138042 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138053 5144 cpu_manager.go:414] "containerMap: removing container" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138060 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138072 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138079 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138090 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138097 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138110 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138116 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138131 5144 cpu_manager.go:414] "containerMap: removing container" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138138 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138146 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138153 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138172 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138179 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138189 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138195 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138209 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138215 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138229 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138236 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138254 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138260 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138269 5144 cpu_manager.go:414] "containerMap: removing container" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138275 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138288 5144 cpu_manager.go:414] "containerMap: removing container" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138295 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138490 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138506 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138518 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138528 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138538 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138547 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138559 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138569 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138579 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138587 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138595 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138603 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138613 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138621 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138653 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="276cb1a0-aed3-4155-b71a-0c246b4ff76a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138664 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138672 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138682 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138691 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138702 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da6cba9-2cc3-4503-9ef5-73ec4b285571" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138715 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138725 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138736 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138745 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138770 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="38e39f7c-8d20-4a0a-b6ab-6bdd0ae03fcd" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138777 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:15:49.139024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.138786 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:15:49.142098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.142079 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.197823 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.197795 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb"] Nov 03 01:15:49.228358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.228324 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvtp9\" (UniqueName: \"kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.228536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.228495 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.228776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.228754 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.242303 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.242279 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:15:49.242459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.242392 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:15:49.329952 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.329913 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.330131 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.330012 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvtp9\" (UniqueName: \"kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.330131 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.330120 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.330625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.330591 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.332785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.332760 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.387612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.387540 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvtp9\" (UniqueName: \"kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9\") pod \"maas-api-revoke-sub-56536685-vjmzb\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.459684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.459617 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:49.843223 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:15:49.843190 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5225514_5955_43cb_9550_f4ef20aee65e.slice/crio-01ca21876266ba61e2a5394af4a1367988efd5de5d62d669ad157038968c15e1 WatchSource:0}: Error finding container 01ca21876266ba61e2a5394af4a1367988efd5de5d62d669ad157038968c15e1: Status 404 returned error can't find the container with id 01ca21876266ba61e2a5394af4a1367988efd5de5d62d669ad157038968c15e1 Nov 03 01:15:49.844482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:49.844198 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb"] Nov 03 01:15:50.835455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:50.835410 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" event={"ID":"e5225514-5955-43cb-9550-f4ef20aee65e","Type":"ContainerStarted","Data":"d35c0b689aa208f90c1d0f3e415973c55c50d6c155d9d3b52b5b534a4e765aee"} Nov 03 01:15:50.835986 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:50.835469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" event={"ID":"e5225514-5955-43cb-9550-f4ef20aee65e","Type":"ContainerStarted","Data":"01ca21876266ba61e2a5394af4a1367988efd5de5d62d669ad157038968c15e1"} Nov 03 01:15:50.948679 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:50.948597 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" podStartSLOduration=1.9485830370000001 podStartE2EDuration="1.948583037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:50.94695758 +0000 UTC m=+4241.092652039" watchObservedRunningTime="2026-11-03 01:15:50.948583037 +0000 UTC m=+4241.094277557" Nov 03 01:15:51.847791 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:51.847758 5144 generic.go:356] "Generic (PLEG): container finished" podID="e5225514-5955-43cb-9550-f4ef20aee65e" containerID="d35c0b689aa208f90c1d0f3e415973c55c50d6c155d9d3b52b5b534a4e765aee" exitCode=0 Nov 03 01:15:51.848309 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:51.847848 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" event={"ID":"e5225514-5955-43cb-9550-f4ef20aee65e","Type":"ContainerDied","Data":"d35c0b689aa208f90c1d0f3e415973c55c50d6c155d9d3b52b5b534a4e765aee"} Nov 03 01:15:53.040781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.040745 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:53.193734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.193651 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca\") pod \"e5225514-5955-43cb-9550-f4ef20aee65e\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " Nov 03 01:15:53.193734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.193695 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca\") pod \"e5225514-5955-43cb-9550-f4ef20aee65e\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " Nov 03 01:15:53.193977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.193749 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9\" (UniqueName: \"kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9\") pod \"e5225514-5955-43cb-9550-f4ef20aee65e\" (UID: \"e5225514-5955-43cb-9550-f4ef20aee65e\") " Nov 03 01:15:53.194146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.194116 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca" pod "e5225514-5955-43cb-9550-f4ef20aee65e" (UID: "e5225514-5955-43cb-9550-f4ef20aee65e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:15:53.194837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.194808 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:53.196214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.196184 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca" pod "e5225514-5955-43cb-9550-f4ef20aee65e" (UID: "e5225514-5955-43cb-9550-f4ef20aee65e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:15:53.196325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.196256 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9" pod "e5225514-5955-43cb-9550-f4ef20aee65e" (UID: "e5225514-5955-43cb-9550-f4ef20aee65e"). InnerVolumeSpecName "kube-api-access-xvtp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:15:53.296213 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.296178 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5225514-5955-43cb-9550-f4ef20aee65e-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:53.296213 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.296216 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvtp9\" (UniqueName: \"kubernetes.io/projected/e5225514-5955-43cb-9550-f4ef20aee65e-kube-api-access-xvtp9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:15:53.481932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.481847 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk"] Nov 03 01:15:53.486957 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.486929 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf"] Nov 03 01:15:53.531567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.531532 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw"] Nov 03 01:15:53.867996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.867953 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" Nov 03 01:15:53.867996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.867985 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb" event={"ID":"e5225514-5955-43cb-9550-f4ef20aee65e","Type":"ContainerDied","Data":"01ca21876266ba61e2a5394af4a1367988efd5de5d62d669ad157038968c15e1"} Nov 03 01:15:53.868271 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:15:53.868038 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ca21876266ba61e2a5394af4a1367988efd5de5d62d669ad157038968c15e1" Nov 03 01:16:05.895719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.895686 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm"] Nov 03 01:16:05.896383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896367 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:16:05.896383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896382 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bf46411-92e3-4ed3-a679-970f2ec4107e" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896410 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896415 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edbcba3-16f6-4d8f-8589-ef9dd920dfc1" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896429 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e5225514-5955-43cb-9550-f4ef20aee65e" containerName="revoke-keys" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896434 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5225514-5955-43cb-9550-f4ef20aee65e" containerName="revoke-keys" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896441 5144 cpu_manager.go:414] "containerMap: removing container" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896446 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896453 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896457 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe0e82d-342d-431b-b8bc-a4bd93fc2a09" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896468 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896472 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896485 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:16:05.896508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896490 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4db75669-cf47-4fa3-b67f-17f4e4a7c70a" containerName="cleanup" Nov 03 01:16:05.896950 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896577 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5225514-5955-43cb-9550-f4ef20aee65e" containerName="revoke-keys" Nov 03 01:16:05.896950 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896586 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e7f76d-a799-4120-99a7-f19299a11e72" containerName="cleanup" Nov 03 01:16:05.896950 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896596 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b30373b8-efc1-491d-b87a-40508b78d75a" containerName="cleanup" Nov 03 01:16:05.896950 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.896603 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7226c1f9-036f-48c1-a2ac-067f7a436331" containerName="cleanup" Nov 03 01:16:05.900427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.900406 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:05.956866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.956835 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm"] Nov 03 01:16:05.961678 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.961616 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:05.961843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.961704 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z9fd\" (UniqueName: \"kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:05.961843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.961768 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:05.966469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:05.966424 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:16:06.001073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.001047 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:16:06.066089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.064223 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z9fd\" (UniqueName: \"kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.066089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.064755 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.066089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.065007 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.066089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.065728 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.071652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.071602 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.121438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.121407 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z9fd\" (UniqueName: \"kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd\") pod \"maas-api-revoke-sub-29661110-5pwnm\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.214293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.214201 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:06.596954 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.596920 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm"] Nov 03 01:16:06.597086 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:16:06.597055 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2113b172_36c2_4b09_8a58_7b68ac100390.slice/crio-758036fc523839fe0f02152458b105df1562f866933aa281b450c96dbc220995 WatchSource:0}: Error finding container 758036fc523839fe0f02152458b105df1562f866933aa281b450c96dbc220995: Status 404 returned error can't find the container with id 758036fc523839fe0f02152458b105df1562f866933aa281b450c96dbc220995 Nov 03 01:16:06.986674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:06.986561 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" event={"ID":"2113b172-36c2-4b09-8a58-7b68ac100390","Type":"ContainerStarted","Data":"758036fc523839fe0f02152458b105df1562f866933aa281b450c96dbc220995"} Nov 03 01:16:07.997016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:07.996980 5144 generic.go:356] "Generic (PLEG): container finished" podID="2113b172-36c2-4b09-8a58-7b68ac100390" containerID="976b0e537b59f74507c9c42d2cbecba9c7d314e9a563ed3f96fb771a6ac84301" exitCode=0 Nov 03 01:16:07.998283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:07.998260 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" event={"ID":"2113b172-36c2-4b09-8a58-7b68ac100390","Type":"ContainerDied","Data":"976b0e537b59f74507c9c42d2cbecba9c7d314e9a563ed3f96fb771a6ac84301"} Nov 03 01:16:08.510501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.510457 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" containerID="cri-o://92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6" gracePeriod=45 Nov 03 01:16:08.515472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.515127 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" containerID="cri-o://834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02" gracePeriod=45 Nov 03 01:16:08.557679 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.557493 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" containerID="cri-o://677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e" gracePeriod=45 Nov 03 01:16:08.889306 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.889278 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 01:16:08.893441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.893414 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 01:16:08.897687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.897664 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 01:16:08.910093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910066 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir\") pod \"474d0586-dcfe-478c-ae69-4baaaaeb0766\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " Nov 03 01:16:08.910201 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910134 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs\" (UniqueName: \"kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs\") pod \"474d0586-dcfe-478c-ae69-4baaaaeb0766\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " Nov 03 01:16:08.910269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910256 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk\" (UniqueName: \"kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk\") pod \"474d0586-dcfe-478c-ae69-4baaaaeb0766\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " Nov 03 01:16:08.910313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910292 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home\") pod \"474d0586-dcfe-478c-ae69-4baaaaeb0766\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " Nov 03 01:16:08.910396 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910363 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir" pod "474d0586-dcfe-478c-ae69-4baaaaeb0766" (UID: "474d0586-dcfe-478c-ae69-4baaaaeb0766"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.910483 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910452 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm\") pod \"474d0586-dcfe-478c-ae69-4baaaaeb0766\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " Nov 03 01:16:08.910665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910596 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home" pod "474d0586-dcfe-478c-ae69-4baaaaeb0766" (UID: "474d0586-dcfe-478c-ae69-4baaaaeb0766"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.910826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.910706 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache\") pod \"474d0586-dcfe-478c-ae69-4baaaaeb0766\" (UID: \"474d0586-dcfe-478c-ae69-4baaaaeb0766\") " Nov 03 01:16:08.911144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.911118 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache" pod "474d0586-dcfe-478c-ae69-4baaaaeb0766" (UID: "474d0586-dcfe-478c-ae69-4baaaaeb0766"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.911216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.911151 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.911216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.911165 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:08.912737 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.912715 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs" pod "474d0586-dcfe-478c-ae69-4baaaaeb0766" (UID: "474d0586-dcfe-478c-ae69-4baaaaeb0766"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:08.913169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.913143 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm" pod "474d0586-dcfe-478c-ae69-4baaaaeb0766" (UID: "474d0586-dcfe-478c-ae69-4baaaaeb0766"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:08.913255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:08.913186 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk" pod "474d0586-dcfe-478c-ae69-4baaaaeb0766" (UID: "474d0586-dcfe-478c-ae69-4baaaaeb0766"). InnerVolumeSpecName "kube-api-access-jjhqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.014142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.014103 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir\") pod \"2513fc19-80f9-4ea6-b898-09c3a56164e0\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " Nov 03 01:16:09.015219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.015179 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache\") pod \"067644a1-d5bd-4cb2-a515-f73083ad711e\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " Nov 03 01:16:09.015686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.015667 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache\") pod \"2513fc19-80f9-4ea6-b898-09c3a56164e0\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " Nov 03 01:16:09.015941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.015926 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs\" (UniqueName: \"kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs\") pod \"2513fc19-80f9-4ea6-b898-09c3a56164e0\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " Nov 03 01:16:09.016132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.016102 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm\") pod \"067644a1-d5bd-4cb2-a515-f73083ad711e\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " Nov 03 01:16:09.016339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.016323 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm\") pod \"2513fc19-80f9-4ea6-b898-09c3a56164e0\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " Nov 03 01:16:09.016490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.016475 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk\" (UniqueName: \"kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk\") pod \"067644a1-d5bd-4cb2-a515-f73083ad711e\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " Nov 03 01:16:09.016673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.016659 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home\") pod \"067644a1-d5bd-4cb2-a515-f73083ad711e\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " Nov 03 01:16:09.016814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.016801 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir\") pod \"067644a1-d5bd-4cb2-a515-f73083ad711e\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " Nov 03 01:16:09.016998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.016984 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4\" (UniqueName: \"kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4\") pod \"2513fc19-80f9-4ea6-b898-09c3a56164e0\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " Nov 03 01:16:09.017159 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.017134 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs\" (UniqueName: \"kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs\") pod \"067644a1-d5bd-4cb2-a515-f73083ad711e\" (UID: \"067644a1-d5bd-4cb2-a515-f73083ad711e\") " Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.022437 5144 generic.go:356] "Generic (PLEG): container finished" podID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerID="677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e" exitCode=0 Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.022468 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" event={"ID":"067644a1-d5bd-4cb2-a515-f73083ad711e","Type":"ContainerDied","Data":"677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e"} Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.022568 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.031606 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw" event={"ID":"067644a1-d5bd-4cb2-a515-f73083ad711e","Type":"ContainerDied","Data":"da5f867ad61c29ad92764a5325ea59a1ad3e8a04c5916d28e00312c3e5855d6e"} Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.031468 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home\") pod \"2513fc19-80f9-4ea6-b898-09c3a56164e0\" (UID: \"2513fc19-80f9-4ea6-b898-09c3a56164e0\") " Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.014797 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir" pod "2513fc19-80f9-4ea6-b898-09c3a56164e0" (UID: "2513fc19-80f9-4ea6-b898-09c3a56164e0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.024212 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs" pod "067644a1-d5bd-4cb2-a515-f73083ad711e" (UID: "067644a1-d5bd-4cb2-a515-f73083ad711e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.024438 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home" pod "067644a1-d5bd-4cb2-a515-f73083ad711e" (UID: "067644a1-d5bd-4cb2-a515-f73083ad711e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.024618 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir" pod "067644a1-d5bd-4cb2-a515-f73083ad711e" (UID: "067644a1-d5bd-4cb2-a515-f73083ad711e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.025338 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs" pod "2513fc19-80f9-4ea6-b898-09c3a56164e0" (UID: "2513fc19-80f9-4ea6-b898-09c3a56164e0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.025577 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache" pod "067644a1-d5bd-4cb2-a515-f73083ad711e" (UID: "067644a1-d5bd-4cb2-a515-f73083ad711e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.031914 5144 scope.go:122] "RemoveContainer" containerID="677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.025798 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache" pod "2513fc19-80f9-4ea6-b898-09c3a56164e0" (UID: "2513fc19-80f9-4ea6-b898-09c3a56164e0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.027313 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk" pod "067644a1-d5bd-4cb2-a515-f73083ad711e" (UID: "067644a1-d5bd-4cb2-a515-f73083ad711e"). InnerVolumeSpecName "kube-api-access-5mmtk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.031511 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4" pod "2513fc19-80f9-4ea6-b898-09c3a56164e0" (UID: "2513fc19-80f9-4ea6-b898-09c3a56164e0"). InnerVolumeSpecName "kube-api-access-2nck4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.031748 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home" pod "2513fc19-80f9-4ea6-b898-09c3a56164e0" (UID: "2513fc19-80f9-4ea6-b898-09c3a56164e0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033288 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/067644a1-d5bd-4cb2-a515-f73083ad711e-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033313 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033330 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhqk\" (UniqueName: \"kubernetes.io/projected/474d0586-dcfe-478c-ae69-4baaaaeb0766-kube-api-access-jjhqk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033346 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033360 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033381 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033395 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033409 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2513fc19-80f9-4ea6-b898-09c3a56164e0-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033422 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/474d0586-dcfe-478c-ae69-4baaaaeb0766-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033435 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mmtk\" (UniqueName: \"kubernetes.io/projected/067644a1-d5bd-4cb2-a515-f73083ad711e-kube-api-access-5mmtk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033449 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033461 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033474 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/474d0586-dcfe-478c-ae69-4baaaaeb0766-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.033487 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nck4\" (UniqueName: \"kubernetes.io/projected/2513fc19-80f9-4ea6-b898-09c3a56164e0-kube-api-access-2nck4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.034397 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm" pod "2513fc19-80f9-4ea6-b898-09c3a56164e0" (UID: "2513fc19-80f9-4ea6-b898-09c3a56164e0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.042204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.034417 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm" pod "067644a1-d5bd-4cb2-a515-f73083ad711e" (UID: "067644a1-d5bd-4cb2-a515-f73083ad711e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:16:09.045063 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.045022 5144 generic.go:356] "Generic (PLEG): container finished" podID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerID="92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6" exitCode=0 Nov 03 01:16:09.045222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.045131 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" event={"ID":"474d0586-dcfe-478c-ae69-4baaaaeb0766","Type":"ContainerDied","Data":"92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6"} Nov 03 01:16:09.045222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.045175 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" event={"ID":"474d0586-dcfe-478c-ae69-4baaaaeb0766","Type":"ContainerDied","Data":"540270eb212bfd4ce159ea01442dfbe3651c96675b9c6923680d6be14abec908"} Nov 03 01:16:09.045355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.045236 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf" Nov 03 01:16:09.052937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.047641 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" Nov 03 01:16:09.052937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.047685 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" event={"ID":"2513fc19-80f9-4ea6-b898-09c3a56164e0","Type":"ContainerDied","Data":"834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02"} Nov 03 01:16:09.052937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.047523 5144 generic.go:356] "Generic (PLEG): container finished" podID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerID="834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02" exitCode=0 Nov 03 01:16:09.052937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.052578 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" event={"ID":"2513fc19-80f9-4ea6-b898-09c3a56164e0","Type":"ContainerDied","Data":"6c9c03cf66f9b40276b13c21ebf1bfb1bd9d59f7131eccec8585887faeeac0c2"} Nov 03 01:16:09.063858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.063839 5144 scope.go:122] "RemoveContainer" containerID="677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e" Nov 03 01:16:09.064384 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:16:09.064359 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e\": container with ID starting with 677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e not found: ID does not exist" containerID="677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e" Nov 03 01:16:09.064535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.064510 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e"} err="failed to get container status \"677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e\": rpc error: code = NotFound desc = could not find container \"677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e\": container with ID starting with 677a709217800b144f482614a4e7fa84a8939117922a374b480c5caddd3c3d5e not found: ID does not exist" Nov 03 01:16:09.065272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.065256 5144 scope.go:122] "RemoveContainer" containerID="92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6" Nov 03 01:16:09.088315 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.088284 5144 scope.go:122] "RemoveContainer" containerID="92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6" Nov 03 01:16:09.088819 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:16:09.088795 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6\": container with ID starting with 92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6 not found: ID does not exist" containerID="92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6" Nov 03 01:16:09.088931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.088834 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6"} err="failed to get container status \"92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6\": rpc error: code = NotFound desc = could not find container \"92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6\": container with ID starting with 92d406af75e238e7970e0da8b5a42fa49cf19f8ab35ff52ca0f85c71976106a6 not found: ID does not exist" Nov 03 01:16:09.088931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.088861 5144 scope.go:122] "RemoveContainer" containerID="834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02" Nov 03 01:16:09.101783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.101739 5144 scope.go:122] "RemoveContainer" containerID="834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02" Nov 03 01:16:09.102233 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:16:09.102211 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02\": container with ID starting with 834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02 not found: ID does not exist" containerID="834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02" Nov 03 01:16:09.102355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.102239 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02"} err="failed to get container status \"834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02\": rpc error: code = NotFound desc = could not find container \"834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02\": container with ID starting with 834830544eb928490ac5d991de49ac35a90837d222dc1bcd9f82719773e5df02 not found: ID does not exist" Nov 03 01:16:09.134752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.134719 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/067644a1-d5bd-4cb2-a515-f73083ad711e-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.134752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.134753 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2513fc19-80f9-4ea6-b898-09c3a56164e0-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.231750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.231714 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:09.280983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.280943 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26"] Nov 03 01:16:09.336747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.336653 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd\" (UniqueName: \"kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd\") pod \"2113b172-36c2-4b09-8a58-7b68ac100390\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " Nov 03 01:16:09.336747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.336716 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca\") pod \"2113b172-36c2-4b09-8a58-7b68ac100390\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " Nov 03 01:16:09.336989 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.336784 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca\") pod \"2113b172-36c2-4b09-8a58-7b68ac100390\" (UID: \"2113b172-36c2-4b09-8a58-7b68ac100390\") " Nov 03 01:16:09.337107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.337080 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca" pod "2113b172-36c2-4b09-8a58-7b68ac100390" (UID: "2113b172-36c2-4b09-8a58-7b68ac100390"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:09.337351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.337328 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.339254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.339225 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca" pod "2113b172-36c2-4b09-8a58-7b68ac100390" (UID: "2113b172-36c2-4b09-8a58-7b68ac100390"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:09.339546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.339475 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd" pod "2113b172-36c2-4b09-8a58-7b68ac100390" (UID: "2113b172-36c2-4b09-8a58-7b68ac100390"). InnerVolumeSpecName "kube-api-access-4z9fd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:09.351401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.351369 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk"] Nov 03 01:16:09.355128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.355101 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk"] Nov 03 01:16:09.438587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.438551 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z9fd\" (UniqueName: \"kubernetes.io/projected/2113b172-36c2-4b09-8a58-7b68ac100390-kube-api-access-4z9fd\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.438587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.438585 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2113b172-36c2-4b09-8a58-7b68ac100390-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:09.606663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.606556 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf"] Nov 03 01:16:09.622364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.622317 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/distinct-w0-ebeeb9-kserve-65d8b5ff8f-sh2bf"] Nov 03 01:16:09.786143 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.786101 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw"] Nov 03 01:16:09.805100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.805065 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-ebeeb9/simulator-w0-ebeeb9-kserve-5d988bbcb7-7vzzw"] Nov 03 01:16:09.866829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.866727 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-ebeeb9/distinct-2-w0-ebeeb9-kserve-c4767cb54-87bgk" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" probeResult="failure" output="Get \"https://10.132.0.101:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 01:16:09.997180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.997149 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" path="/var/lib/kubelet/pods/067644a1-d5bd-4cb2-a515-f73083ad711e/volumes" Nov 03 01:16:09.997597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.997584 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" path="/var/lib/kubelet/pods/2513fc19-80f9-4ea6-b898-09c3a56164e0/volumes" Nov 03 01:16:09.998019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:09.998005 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" path="/var/lib/kubelet/pods/474d0586-dcfe-478c-ae69-4baaaaeb0766/volumes" Nov 03 01:16:10.064482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:10.064453 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" Nov 03 01:16:10.064955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:10.064450 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm" event={"ID":"2113b172-36c2-4b09-8a58-7b68ac100390","Type":"ContainerDied","Data":"758036fc523839fe0f02152458b105df1562f866933aa281b450c96dbc220995"} Nov 03 01:16:10.064955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:10.064575 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="758036fc523839fe0f02152458b105df1562f866933aa281b450c96dbc220995" Nov 03 01:16:12.158026 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:16:12.157985 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:16:13.719475 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:13.719441 5144 scope.go:122] "RemoveContainer" containerID="538e427cfe45f6b1c01a96d8c3516bdc95607cd1dbfd50fb6cbec98d185c7d7d" Nov 03 01:16:13.730928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:13.730882 5144 scope.go:122] "RemoveContainer" containerID="b9a10c1c0f20ea89ae133636b37393e04e5c07da664e12b395259b6993e75c07" Nov 03 01:16:14.281799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:14.281735 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerName="payload-pre-processing" containerID="cri-o://d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547" gracePeriod=25 Nov 03 01:16:16.735583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:16.735548 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:26.736403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:26.736364 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:16:34.599674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.599643 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 01:16:34.695050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.695014 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq\" (UniqueName: \"kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq\") pod \"36fc44aa-9b16-4134-b5ce-43105372c5e1\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " Nov 03 01:16:34.695232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.695194 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume\") pod \"36fc44aa-9b16-4134-b5ce-43105372c5e1\" (UID: \"36fc44aa-9b16-4134-b5ce-43105372c5e1\") " Nov 03 01:16:34.695563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.695539 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume" pod "36fc44aa-9b16-4134-b5ce-43105372c5e1" (UID: "36fc44aa-9b16-4134-b5ce-43105372c5e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:34.695848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.695768 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36fc44aa-9b16-4134-b5ce-43105372c5e1-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:34.697411 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.697382 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq" pod "36fc44aa-9b16-4134-b5ce-43105372c5e1" (UID: "36fc44aa-9b16-4134-b5ce-43105372c5e1"). InnerVolumeSpecName "kube-api-access-nn5dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:34.797283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:34.797244 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nn5dq\" (UniqueName: \"kubernetes.io/projected/36fc44aa-9b16-4134-b5ce-43105372c5e1-kube-api-access-nn5dq\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:35.282998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.282958 5144 generic.go:356] "Generic (PLEG): container finished" podID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerID="d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547" exitCode=0 Nov 03 01:16:35.283188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.283011 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" Nov 03 01:16:35.283188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.283046 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" event={"ID":"36fc44aa-9b16-4134-b5ce-43105372c5e1","Type":"ContainerDied","Data":"d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547"} Nov 03 01:16:35.283188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.283103 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26" event={"ID":"36fc44aa-9b16-4134-b5ce-43105372c5e1","Type":"ContainerDied","Data":"6222045f21e4b41d28763395527414a12a3e7bbc62368d9f5a721c667e1f0302"} Nov 03 01:16:35.283188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.283125 5144 scope.go:122] "RemoveContainer" containerID="d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547" Nov 03 01:16:35.294082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.294056 5144 scope.go:122] "RemoveContainer" containerID="d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547" Nov 03 01:16:35.294428 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:16:35.294407 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547\": container with ID starting with d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547 not found: ID does not exist" containerID="d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547" Nov 03 01:16:35.294483 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.294438 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547"} err="failed to get container status \"d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547\": rpc error: code = NotFound desc = could not find container \"d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547\": container with ID starting with d64314d995b5409d3675cb8fe1387292400b03b3f58b3566bda9452b9bcda547 not found: ID does not exist" Nov 03 01:16:35.446871 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.446832 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26"] Nov 03 01:16:35.453216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.453184 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-e4c8b1-7f57fdc57f-dxc26"] Nov 03 01:16:35.999460 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:35.999413 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" path="/var/lib/kubelet/pods/36fc44aa-9b16-4134-b5ce-43105372c5e1/volumes" Nov 03 01:16:42.158139 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:16:42.158099 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:16:46.054338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:46.054297 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q"] Nov 03 01:16:46.103908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:46.103875 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dd1f35c3-5g26q"] Nov 03 01:16:47.997925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:47.997891 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b31a38c1-07e7-46ba-9c2d-753df766b600" path="/var/lib/kubelet/pods/b31a38c1-07e7-46ba-9c2d-753df766b600/volumes" Nov 03 01:16:52.169542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.169504 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp"] Nov 03 01:16:52.170217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170200 5144 cpu_manager.go:414] "containerMap: removing container" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerName="payload-pre-processing" Nov 03 01:16:52.170268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170219 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerName="payload-pre-processing" Nov 03 01:16:52.170268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170240 5144 cpu_manager.go:414] "containerMap: removing container" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" Nov 03 01:16:52.170268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170245 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" Nov 03 01:16:52.170400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170282 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" Nov 03 01:16:52.170400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170287 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" Nov 03 01:16:52.170400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170296 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2113b172-36c2-4b09-8a58-7b68ac100390" containerName="revoke-keys" Nov 03 01:16:52.170400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170300 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2113b172-36c2-4b09-8a58-7b68ac100390" containerName="revoke-keys" Nov 03 01:16:52.170400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170314 5144 cpu_manager.go:414] "containerMap: removing container" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" Nov 03 01:16:52.170400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170318 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" Nov 03 01:16:52.170590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170433 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2513fc19-80f9-4ea6-b898-09c3a56164e0" containerName="main" Nov 03 01:16:52.170590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170444 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2113b172-36c2-4b09-8a58-7b68ac100390" containerName="revoke-keys" Nov 03 01:16:52.170590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170451 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="067644a1-d5bd-4cb2-a515-f73083ad711e" containerName="main" Nov 03 01:16:52.170590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170455 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="36fc44aa-9b16-4134-b5ce-43105372c5e1" containerName="payload-pre-processing" Nov 03 01:16:52.170590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.170462 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="474d0586-dcfe-478c-ae69-4baaaaeb0766" containerName="main" Nov 03 01:16:52.174180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.174164 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.230011 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.229979 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp"] Nov 03 01:16:52.262883 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.262851 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.263080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.262923 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6smkj\" (UniqueName: \"kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.263241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.263210 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.273788 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.273763 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:16:52.274060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.274043 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:16:52.364031 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.363992 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.364239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.364064 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6smkj\" (UniqueName: \"kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.364239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.364224 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.364920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.364897 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.366610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.366592 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.421596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.421512 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6smkj\" (UniqueName: \"kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj\") pod \"maas-api-revoke-sub-7effafee-vmptp\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.488198 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.488163 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:52.869064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:52.869032 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp"] Nov 03 01:16:52.870581 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:16:52.870553 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba88532_0bc8_4f2b_bb18_63e016f1aeb8.slice/crio-18f682a7bbce4a27244c58e6bc5215b41de0149b6d8a9fc48f4534983cf74c1a WatchSource:0}: Error finding container 18f682a7bbce4a27244c58e6bc5215b41de0149b6d8a9fc48f4534983cf74c1a: Status 404 returned error can't find the container with id 18f682a7bbce4a27244c58e6bc5215b41de0149b6d8a9fc48f4534983cf74c1a Nov 03 01:16:53.439402 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:53.439307 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" event={"ID":"bba88532-0bc8-4f2b-bb18-63e016f1aeb8","Type":"ContainerStarted","Data":"1f7d6b748d2648449a3765b1492f47cd35a5a69ff89751982dd22716d9d6bafc"} Nov 03 01:16:53.439402 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:53.439363 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" event={"ID":"bba88532-0bc8-4f2b-bb18-63e016f1aeb8","Type":"ContainerStarted","Data":"18f682a7bbce4a27244c58e6bc5215b41de0149b6d8a9fc48f4534983cf74c1a"} Nov 03 01:16:53.554300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:53.554234 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" podStartSLOduration=1.554217649 podStartE2EDuration="1.554217649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:16:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:16:53.552020949 +0000 UTC m=+4303.697715424" watchObservedRunningTime="2026-11-03 01:16:53.554217649 +0000 UTC m=+4303.699912105" Nov 03 01:16:54.450889 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:54.450857 5144 generic.go:356] "Generic (PLEG): container finished" podID="bba88532-0bc8-4f2b-bb18-63e016f1aeb8" containerID="1f7d6b748d2648449a3765b1492f47cd35a5a69ff89751982dd22716d9d6bafc" exitCode=0 Nov 03 01:16:54.451339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:54.450951 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" event={"ID":"bba88532-0bc8-4f2b-bb18-63e016f1aeb8","Type":"ContainerDied","Data":"1f7d6b748d2648449a3765b1492f47cd35a5a69ff89751982dd22716d9d6bafc"} Nov 03 01:16:55.638969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.638942 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:55.706347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.706311 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca\") pod \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " Nov 03 01:16:55.706577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.706441 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj\" (UniqueName: \"kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj\") pod \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " Nov 03 01:16:55.706577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.706516 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca\") pod \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\" (UID: \"bba88532-0bc8-4f2b-bb18-63e016f1aeb8\") " Nov 03 01:16:55.706871 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.706846 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca" pod "bba88532-0bc8-4f2b-bb18-63e016f1aeb8" (UID: "bba88532-0bc8-4f2b-bb18-63e016f1aeb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:16:55.707054 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.707039 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:55.708829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.708798 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca" pod "bba88532-0bc8-4f2b-bb18-63e016f1aeb8" (UID: "bba88532-0bc8-4f2b-bb18-63e016f1aeb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:16:55.708829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.708806 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj" pod "bba88532-0bc8-4f2b-bb18-63e016f1aeb8" (UID: "bba88532-0bc8-4f2b-bb18-63e016f1aeb8"). InnerVolumeSpecName "kube-api-access-6smkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:16:55.808387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.808352 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6smkj\" (UniqueName: \"kubernetes.io/projected/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-kube-api-access-6smkj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:55.808387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:55.808384 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bba88532-0bc8-4f2b-bb18-63e016f1aeb8-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:16:56.469068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:56.468993 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" Nov 03 01:16:56.469068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:56.469010 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp" event={"ID":"bba88532-0bc8-4f2b-bb18-63e016f1aeb8","Type":"ContainerDied","Data":"18f682a7bbce4a27244c58e6bc5215b41de0149b6d8a9fc48f4534983cf74c1a"} Nov 03 01:16:56.469291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:16:56.469094 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18f682a7bbce4a27244c58e6bc5215b41de0149b6d8a9fc48f4534983cf74c1a" Nov 03 01:17:02.945110 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:02.945077 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp"] Nov 03 01:17:02.945972 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:02.945951 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bba88532-0bc8-4f2b-bb18-63e016f1aeb8" containerName="revoke-keys" Nov 03 01:17:02.945972 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:02.945974 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba88532-0bc8-4f2b-bb18-63e016f1aeb8" containerName="revoke-keys" Nov 03 01:17:02.946219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:02.946204 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba88532-0bc8-4f2b-bb18-63e016f1aeb8" containerName="revoke-keys" Nov 03 01:17:02.950082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:02.950063 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.002447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.002414 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:17:03.051904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.051873 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:17:03.056403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.056377 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp"] Nov 03 01:17:03.107926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.107888 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.108089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.107976 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.108089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.108047 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrjg\" (UniqueName: \"kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.210091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.210001 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.210091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.210054 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.210091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.210086 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrjg\" (UniqueName: \"kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.210868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.210842 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.212644 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.212607 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.267336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.267298 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrjg\" (UniqueName: \"kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg\") pod \"maas-api-revoke-sub-0bafb469-bdkwp\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.563688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.563612 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:03.955387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:03.955357 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp"] Nov 03 01:17:03.957225 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:17:03.957195 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd4b33e_cf3b_4323_a101_085bfc19239f.slice/crio-fdc7a7133fb03ae825876e351165b80e22d2f80ce4c6e8c0918760b31067df63 WatchSource:0}: Error finding container fdc7a7133fb03ae825876e351165b80e22d2f80ce4c6e8c0918760b31067df63: Status 404 returned error can't find the container with id fdc7a7133fb03ae825876e351165b80e22d2f80ce4c6e8c0918760b31067df63 Nov 03 01:17:04.538549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:04.538516 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" event={"ID":"8cd4b33e-cf3b-4323-a101-085bfc19239f","Type":"ContainerStarted","Data":"e7bf0d0bcf4222195f84b7ba46f8704fa28c56304b25207b9f5568792089b249"} Nov 03 01:17:04.538793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:04.538579 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" event={"ID":"8cd4b33e-cf3b-4323-a101-085bfc19239f","Type":"ContainerStarted","Data":"fdc7a7133fb03ae825876e351165b80e22d2f80ce4c6e8c0918760b31067df63"} Nov 03 01:17:04.654556 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:04.654501 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" podStartSLOduration=2.6544855739999997 podStartE2EDuration="2.654485574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:04.652482681 +0000 UTC m=+4314.798177153" watchObservedRunningTime="2026-11-03 01:17:04.654485574 +0000 UTC m=+4314.800180030" Nov 03 01:17:05.548912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:05.548879 5144 generic.go:356] "Generic (PLEG): container finished" podID="8cd4b33e-cf3b-4323-a101-085bfc19239f" containerID="e7bf0d0bcf4222195f84b7ba46f8704fa28c56304b25207b9f5568792089b249" exitCode=0 Nov 03 01:17:05.549369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:05.548966 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" event={"ID":"8cd4b33e-cf3b-4323-a101-085bfc19239f","Type":"ContainerDied","Data":"e7bf0d0bcf4222195f84b7ba46f8704fa28c56304b25207b9f5568792089b249"} Nov 03 01:17:06.730948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.730918 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:06.857349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.857260 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg\" (UniqueName: \"kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg\") pod \"8cd4b33e-cf3b-4323-a101-085bfc19239f\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " Nov 03 01:17:06.857538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.857377 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca\") pod \"8cd4b33e-cf3b-4323-a101-085bfc19239f\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " Nov 03 01:17:06.857612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.857557 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca\") pod \"8cd4b33e-cf3b-4323-a101-085bfc19239f\" (UID: \"8cd4b33e-cf3b-4323-a101-085bfc19239f\") " Nov 03 01:17:06.868585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.857988 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca" pod "8cd4b33e-cf3b-4323-a101-085bfc19239f" (UID: "8cd4b33e-cf3b-4323-a101-085bfc19239f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:06.868585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.861711 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg" pod "8cd4b33e-cf3b-4323-a101-085bfc19239f" (UID: "8cd4b33e-cf3b-4323-a101-085bfc19239f"). InnerVolumeSpecName "kube-api-access-pmrjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:06.868585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.861763 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca" pod "8cd4b33e-cf3b-4323-a101-085bfc19239f" (UID: "8cd4b33e-cf3b-4323-a101-085bfc19239f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:06.959005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.958967 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmrjg\" (UniqueName: \"kubernetes.io/projected/8cd4b33e-cf3b-4323-a101-085bfc19239f-kube-api-access-pmrjg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:06.959005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.959000 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:06.959005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:06.959010 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cd4b33e-cf3b-4323-a101-085bfc19239f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:07.567675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:07.567623 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" event={"ID":"8cd4b33e-cf3b-4323-a101-085bfc19239f","Type":"ContainerDied","Data":"fdc7a7133fb03ae825876e351165b80e22d2f80ce4c6e8c0918760b31067df63"} Nov 03 01:17:07.567864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:07.567695 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdc7a7133fb03ae825876e351165b80e22d2f80ce4c6e8c0918760b31067df63" Nov 03 01:17:07.567864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:07.567651 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp" Nov 03 01:17:12.159051 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:17:12.159004 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:17:14.569814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.569775 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954"] Nov 03 01:17:14.570750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.570727 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8cd4b33e-cf3b-4323-a101-085bfc19239f" containerName="revoke-keys" Nov 03 01:17:14.570750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.570751 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cd4b33e-cf3b-4323-a101-085bfc19239f" containerName="revoke-keys" Nov 03 01:17:14.570959 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.570911 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cd4b33e-cf3b-4323-a101-085bfc19239f" containerName="revoke-keys" Nov 03 01:17:14.576059 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.576038 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.631645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.631603 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954"] Nov 03 01:17:14.674969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.674936 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:17:14.675148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.675073 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:17:14.696049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.696018 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.696215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.696164 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.696311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.696293 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpzpb\" (UniqueName: \"kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.797946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.797906 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.798152 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.798036 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.798152 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.798126 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpzpb\" (UniqueName: \"kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.798700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.798663 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.800540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.800523 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.855421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.855344 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpzpb\" (UniqueName: \"kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb\") pod \"maas-api-revoke-sub-793e935c-pz954\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:14.889880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:14.889851 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:15.279503 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:15.279475 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954"] Nov 03 01:17:15.279970 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:17:15.279946 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod021afe20_e654_48e1_8548_0336da129859.slice/crio-7bc4a14a8267ee0ccd9412dfef7d76ded17ec1913fb3165ae5a6df993ca282e2 WatchSource:0}: Error finding container 7bc4a14a8267ee0ccd9412dfef7d76ded17ec1913fb3165ae5a6df993ca282e2: Status 404 returned error can't find the container with id 7bc4a14a8267ee0ccd9412dfef7d76ded17ec1913fb3165ae5a6df993ca282e2 Nov 03 01:17:15.630445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:15.630405 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" event={"ID":"021afe20-e654-48e1-8548-0336da129859","Type":"ContainerStarted","Data":"7bc4a14a8267ee0ccd9412dfef7d76ded17ec1913fb3165ae5a6df993ca282e2"} Nov 03 01:17:16.640087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:16.639996 5144 generic.go:356] "Generic (PLEG): container finished" podID="021afe20-e654-48e1-8548-0336da129859" containerID="7e0b1e821abf99e6e5a00f15e0921e2220647176ffb092ebd4117a8f3b2676db" exitCode=0 Nov 03 01:17:16.640490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:16.640087 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" event={"ID":"021afe20-e654-48e1-8548-0336da129859","Type":"ContainerDied","Data":"7e0b1e821abf99e6e5a00f15e0921e2220647176ffb092ebd4117a8f3b2676db"} Nov 03 01:17:17.804799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.804773 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:17.947661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.947551 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca\") pod \"021afe20-e654-48e1-8548-0336da129859\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " Nov 03 01:17:17.947661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.947609 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb\" (UniqueName: \"kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb\") pod \"021afe20-e654-48e1-8548-0336da129859\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " Nov 03 01:17:17.947964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.947811 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca\") pod \"021afe20-e654-48e1-8548-0336da129859\" (UID: \"021afe20-e654-48e1-8548-0336da129859\") " Nov 03 01:17:17.948757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.948729 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca" pod "021afe20-e654-48e1-8548-0336da129859" (UID: "021afe20-e654-48e1-8548-0336da129859"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:17.951750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.951711 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca" pod "021afe20-e654-48e1-8548-0336da129859" (UID: "021afe20-e654-48e1-8548-0336da129859"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:17.954734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:17.954702 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb" pod "021afe20-e654-48e1-8548-0336da129859" (UID: "021afe20-e654-48e1-8548-0336da129859"). InnerVolumeSpecName "kube-api-access-kpzpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:18.049572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:18.049542 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/021afe20-e654-48e1-8548-0336da129859-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:18.049572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:18.049571 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpzpb\" (UniqueName: \"kubernetes.io/projected/021afe20-e654-48e1-8548-0336da129859-kube-api-access-kpzpb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:18.049572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:18.049581 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/021afe20-e654-48e1-8548-0336da129859-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:18.655753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:18.655712 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" event={"ID":"021afe20-e654-48e1-8548-0336da129859","Type":"ContainerDied","Data":"7bc4a14a8267ee0ccd9412dfef7d76ded17ec1913fb3165ae5a6df993ca282e2"} Nov 03 01:17:18.655753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:18.655747 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954" Nov 03 01:17:18.656001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:18.655777 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bc4a14a8267ee0ccd9412dfef7d76ded17ec1913fb3165ae5a6df993ca282e2" Nov 03 01:17:20.493481 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:20.493447 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv"] Nov 03 01:17:20.493999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:20.493711 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" containerID="cri-o://e0e160d957d14cae904daa34b81356a180d34b3327611d9f131ae3725259bbbd" gracePeriod=30 Nov 03 01:17:23.937748 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:23.937715 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8"] Nov 03 01:17:24.051164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:24.051116 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9"] Nov 03 01:17:25.500674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.500626 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h"] Nov 03 01:17:25.501313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.501298 5144 cpu_manager.go:414] "containerMap: removing container" podUID="021afe20-e654-48e1-8548-0336da129859" containerName="revoke-keys" Nov 03 01:17:25.501369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.501314 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="021afe20-e654-48e1-8548-0336da129859" containerName="revoke-keys" Nov 03 01:17:25.501438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.501429 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="021afe20-e654-48e1-8548-0336da129859" containerName="revoke-keys" Nov 03 01:17:25.505259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.505238 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.569540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.557191 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:17:25.569540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.558828 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:17:25.591897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.591864 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h"] Nov 03 01:17:25.668789 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.668583 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.676660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.672843 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkj9j\" (UniqueName: \"kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.676660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.672951 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.716184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.716146 5144 generic.go:356] "Generic (PLEG): container finished" podID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerID="e0e160d957d14cae904daa34b81356a180d34b3327611d9f131ae3725259bbbd" exitCode=0 Nov 03 01:17:25.716184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.716177 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" event={"ID":"ffd6edfa-28b9-4c2d-8983-377ec06c0e29","Type":"ContainerDied","Data":"e0e160d957d14cae904daa34b81356a180d34b3327611d9f131ae3725259bbbd"} Nov 03 01:17:25.774691 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.774654 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.774920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.774901 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkj9j\" (UniqueName: \"kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.775018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.774986 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.775717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.775690 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.778671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.777769 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.832587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.832561 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkj9j\" (UniqueName: \"kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j\") pod \"maas-api-revoke-sub-e72c737d-rjs6h\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.854722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.854697 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 01:17:25.909971 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.909942 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:25.977150 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977123 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977194 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977256 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977293 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977471 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977329 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977471 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977379 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977471 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977441 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977485 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977599 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data\") pod \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\" (UID: \"ffd6edfa-28b9-4c2d-8983-377ec06c0e29\") " Nov 03 01:17:25.977762 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977718 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:25.978249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.977920 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.978249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978056 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.978249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978156 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.978249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978242 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.978665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978466 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.978665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978487 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.978665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978502 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.978665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978515 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.978665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.978529 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:25.980158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.980133 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "kube-api-access-vlgg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:25.980698 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.980673 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:17:25.983253 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.983230 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:17:25.983339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:25.983278 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token" pod "ffd6edfa-28b9-4c2d-8983-377ec06c0e29" (UID: "ffd6edfa-28b9-4c2d-8983-377ec06c0e29"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:26.046101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.046016 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b"] Nov 03 01:17:26.050940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.050907 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b032cc2f-sv42b"] Nov 03 01:17:26.080232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.080197 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:26.080232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.080224 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:26.080232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.080234 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:26.080476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.080247 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlgg6\" (UniqueName: \"kubernetes.io/projected/ffd6edfa-28b9-4c2d-8983-377ec06c0e29-kube-api-access-vlgg6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:26.289152 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.289124 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h"] Nov 03 01:17:26.290032 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:17:26.290002 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7de899b5_3ced_4c50_9a52_595ef4764915.slice/crio-e26bbe4260facf65984c0d38084b4039ac24c791e83f13b602f08c5002588df8 WatchSource:0}: Error finding container e26bbe4260facf65984c0d38084b4039ac24c791e83f13b602f08c5002588df8: Status 404 returned error can't find the container with id e26bbe4260facf65984c0d38084b4039ac24c791e83f13b602f08c5002588df8 Nov 03 01:17:26.728488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.728456 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" Nov 03 01:17:26.729004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.728491 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv" event={"ID":"ffd6edfa-28b9-4c2d-8983-377ec06c0e29","Type":"ContainerDied","Data":"678b6e1976b37d084ee5c1ce1178a7a5a34f32d6f554a068b166f6b7d37f6f4c"} Nov 03 01:17:26.729004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.728860 5144 scope.go:122] "RemoveContainer" containerID="e0e160d957d14cae904daa34b81356a180d34b3327611d9f131ae3725259bbbd" Nov 03 01:17:26.730731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.730700 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" event={"ID":"7de899b5-3ced-4c50-9a52-595ef4764915","Type":"ContainerStarted","Data":"0539a14369fec689b73776ac3297d355bebc520362b907e467d7af3cc2f0fbe2"} Nov 03 01:17:26.731013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.730758 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" event={"ID":"7de899b5-3ced-4c50-9a52-595ef4764915","Type":"ContainerStarted","Data":"e26bbe4260facf65984c0d38084b4039ac24c791e83f13b602f08c5002588df8"} Nov 03 01:17:26.897156 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:26.897040 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" podStartSLOduration=1.897025591 podStartE2EDuration="1.897025591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:26.893108173 +0000 UTC m=+4337.038802627" watchObservedRunningTime="2026-11-03 01:17:26.897025591 +0000 UTC m=+4337.042720044" Nov 03 01:17:27.007300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:27.007268 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv"] Nov 03 01:17:27.013035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:27.013005 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-ebeeb9-openshift-default-645f8d7477-w4mpv"] Nov 03 01:17:27.742000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:27.741873 5144 generic.go:356] "Generic (PLEG): container finished" podID="7de899b5-3ced-4c50-9a52-595ef4764915" containerID="0539a14369fec689b73776ac3297d355bebc520362b907e467d7af3cc2f0fbe2" exitCode=0 Nov 03 01:17:27.742000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:27.741960 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" event={"ID":"7de899b5-3ced-4c50-9a52-595ef4764915","Type":"ContainerDied","Data":"0539a14369fec689b73776ac3297d355bebc520362b907e467d7af3cc2f0fbe2"} Nov 03 01:17:27.997267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:27.997192 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0aafdda-1fee-4532-aa89-f6af6e1ba6c8" path="/var/lib/kubelet/pods/c0aafdda-1fee-4532-aa89-f6af6e1ba6c8/volumes" Nov 03 01:17:27.997713 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:27.997697 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" path="/var/lib/kubelet/pods/ffd6edfa-28b9-4c2d-8983-377ec06c0e29/volumes" Nov 03 01:17:28.907721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:28.907693 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:28.938641 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:28.938582 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" containerID="cri-o://0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5" gracePeriod=25 Nov 03 01:17:29.018079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.018043 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:17:29.018271 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.018170 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca\") pod \"7de899b5-3ced-4c50-9a52-595ef4764915\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " Nov 03 01:17:29.018271 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.018244 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j\" (UniqueName: \"kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j\") pod \"7de899b5-3ced-4c50-9a52-595ef4764915\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " Nov 03 01:17:29.018448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.018321 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca\") pod \"7de899b5-3ced-4c50-9a52-595ef4764915\" (UID: \"7de899b5-3ced-4c50-9a52-595ef4764915\") " Nov 03 01:17:29.018753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.018720 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca" pod "7de899b5-3ced-4c50-9a52-595ef4764915" (UID: "7de899b5-3ced-4c50-9a52-595ef4764915"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:29.019081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.019056 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:29.021200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.021167 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j" pod "7de899b5-3ced-4c50-9a52-595ef4764915" (UID: "7de899b5-3ced-4c50-9a52-595ef4764915"). InnerVolumeSpecName "kube-api-access-xkj9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:29.021433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.021406 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca" pod "7de899b5-3ced-4c50-9a52-595ef4764915" (UID: "7de899b5-3ced-4c50-9a52-595ef4764915"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:17:29.052326 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.052289 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerName="payload-pre-processing" containerID="cri-o://83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972" gracePeriod=25 Nov 03 01:17:29.119889 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.119862 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7de899b5-3ced-4c50-9a52-595ef4764915-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:29.119889 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.119889 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkj9j\" (UniqueName: \"kubernetes.io/projected/7de899b5-3ced-4c50-9a52-595ef4764915-kube-api-access-xkj9j\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:29.759447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.759297 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" Nov 03 01:17:29.759447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.759337 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h" event={"ID":"7de899b5-3ced-4c50-9a52-595ef4764915","Type":"ContainerDied","Data":"e26bbe4260facf65984c0d38084b4039ac24c791e83f13b602f08c5002588df8"} Nov 03 01:17:29.759752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:29.759442 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e26bbe4260facf65984c0d38084b4039ac24c791e83f13b602f08c5002588df8" Nov 03 01:17:31.204574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:31.204534 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs"] Nov 03 01:17:31.312172 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:31.312139 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt"] Nov 03 01:17:36.204881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.204839 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" podUID="c76fc554-99de-4249-aed2-5e9d99fe9636" containerName="payload-processing" containerID="cri-o://76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3" gracePeriod=25 Nov 03 01:17:36.312867 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.312826 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" podUID="83a0c076-8805-4771-8e54-99f1333aff51" containerName="payload-pre-processing" containerID="cri-o://557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e" gracePeriod=25 Nov 03 01:17:36.496378 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.495966 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:17:36.603203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.603178 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:17:36.631492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.631455 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume\") pod \"c76fc554-99de-4249-aed2-5e9d99fe9636\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " Nov 03 01:17:36.631833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.631809 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd\" (UniqueName: \"kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd\") pod \"c76fc554-99de-4249-aed2-5e9d99fe9636\" (UID: \"c76fc554-99de-4249-aed2-5e9d99fe9636\") " Nov 03 01:17:36.631937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.631887 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume" pod "c76fc554-99de-4249-aed2-5e9d99fe9636" (UID: "c76fc554-99de-4249-aed2-5e9d99fe9636"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:36.632409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.632384 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c76fc554-99de-4249-aed2-5e9d99fe9636-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.634159 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.634118 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd" pod "c76fc554-99de-4249-aed2-5e9d99fe9636" (UID: "c76fc554-99de-4249-aed2-5e9d99fe9636"). InnerVolumeSpecName "kube-api-access-qgrhd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:36.733894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.733816 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47\" (UniqueName: \"kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47\") pod \"83a0c076-8805-4771-8e54-99f1333aff51\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " Nov 03 01:17:36.734084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.734001 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume\") pod \"83a0c076-8805-4771-8e54-99f1333aff51\" (UID: \"83a0c076-8805-4771-8e54-99f1333aff51\") " Nov 03 01:17:36.734522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.734502 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgrhd\" (UniqueName: \"kubernetes.io/projected/c76fc554-99de-4249-aed2-5e9d99fe9636-kube-api-access-qgrhd\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.734884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.734856 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume" pod "83a0c076-8805-4771-8e54-99f1333aff51" (UID: "83a0c076-8805-4771-8e54-99f1333aff51"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:36.736956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.736927 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47" pod "83a0c076-8805-4771-8e54-99f1333aff51" (UID: "83a0c076-8805-4771-8e54-99f1333aff51"). InnerVolumeSpecName "kube-api-access-t6t47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:36.813255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.813215 5144 generic.go:356] "Generic (PLEG): container finished" podID="c76fc554-99de-4249-aed2-5e9d99fe9636" containerID="76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3" exitCode=0 Nov 03 01:17:36.813462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.813265 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" Nov 03 01:17:36.813462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.813297 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" event={"ID":"c76fc554-99de-4249-aed2-5e9d99fe9636","Type":"ContainerDied","Data":"76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3"} Nov 03 01:17:36.813462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.813360 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs" event={"ID":"c76fc554-99de-4249-aed2-5e9d99fe9636","Type":"ContainerDied","Data":"695ff989971c0724c06c2eab6a7afcb8c349113f115d487cc167be6a131c8772"} Nov 03 01:17:36.813462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.813389 5144 scope.go:122] "RemoveContainer" containerID="76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3" Nov 03 01:17:36.818273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.817976 5144 generic.go:356] "Generic (PLEG): container finished" podID="83a0c076-8805-4771-8e54-99f1333aff51" containerID="557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e" exitCode=0 Nov 03 01:17:36.818273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.818031 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" Nov 03 01:17:36.818273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.818103 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" event={"ID":"83a0c076-8805-4771-8e54-99f1333aff51","Type":"ContainerDied","Data":"557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e"} Nov 03 01:17:36.818273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.818158 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt" event={"ID":"83a0c076-8805-4771-8e54-99f1333aff51","Type":"ContainerDied","Data":"971f38799c02043a487c10936c91cd630abefff68e0a15fd88b8e0fd36279463"} Nov 03 01:17:36.830169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.830146 5144 scope.go:122] "RemoveContainer" containerID="76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3" Nov 03 01:17:36.830490 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:17:36.830464 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3\": container with ID starting with 76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3 not found: ID does not exist" containerID="76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3" Nov 03 01:17:36.830596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.830496 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3"} err="failed to get container status \"76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3\": rpc error: code = NotFound desc = could not find container \"76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3\": container with ID starting with 76a61abef0e7104786acc2b2a3814fdd8e7f29b0350b8d760d986b45679324f3 not found: ID does not exist" Nov 03 01:17:36.830596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.830513 5144 scope.go:122] "RemoveContainer" containerID="557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e" Nov 03 01:17:36.836543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.836519 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6t47\" (UniqueName: \"kubernetes.io/projected/83a0c076-8805-4771-8e54-99f1333aff51-kube-api-access-t6t47\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.836696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.836552 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83a0c076-8805-4771-8e54-99f1333aff51-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:36.843149 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.843123 5144 scope.go:122] "RemoveContainer" containerID="557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e" Nov 03 01:17:36.843997 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:17:36.843966 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e\": container with ID starting with 557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e not found: ID does not exist" containerID="557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e" Nov 03 01:17:36.844167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.844134 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e"} err="failed to get container status \"557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e\": rpc error: code = NotFound desc = could not find container \"557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e\": container with ID starting with 557013fd93cb45d566290c48bf1e6c7b436316fe700e311c5110827df1d3005e not found: ID does not exist" Nov 03 01:17:36.978266 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.978235 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs"] Nov 03 01:17:36.982716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:36.982692 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-aee0dd37-b6f4b77-zw7rs"] Nov 03 01:17:37.091380 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:37.091335 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt"] Nov 03 01:17:37.095030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:37.095003 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-aee0dd37-674d947cf-r6tgt"] Nov 03 01:17:38.023740 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:38.023705 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a0c076-8805-4771-8e54-99f1333aff51" path="/var/lib/kubelet/pods/83a0c076-8805-4771-8e54-99f1333aff51/volumes" Nov 03 01:17:38.024390 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:38.024366 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c76fc554-99de-4249-aed2-5e9d99fe9636" path="/var/lib/kubelet/pods/c76fc554-99de-4249-aed2-5e9d99fe9636/volumes" Nov 03 01:17:39.017749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:39.017708 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:17:39.018079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:39.017812 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:17:42.155990 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:17:42.155951 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:17:49.016845 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:17:49.016816 5144 logging.go:55] [core] [Channel #21106 SubChannel #21108]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.245:50052", ServerName: "10.132.0.245:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.245:50052: connect: connection refused" Nov 03 01:17:49.017348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.017322 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:17:49.210500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.210438 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 01:17:49.307948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.307919 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws\" (UniqueName: \"kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws\") pod \"9a14e39a-7816-4289-af4e-a95614befbf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " Nov 03 01:17:49.308079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.308015 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume\") pod \"9a14e39a-7816-4289-af4e-a95614befbf8\" (UID: \"9a14e39a-7816-4289-af4e-a95614befbf8\") " Nov 03 01:17:49.308376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.308354 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume" pod "9a14e39a-7816-4289-af4e-a95614befbf8" (UID: "9a14e39a-7816-4289-af4e-a95614befbf8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:49.310361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.310333 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws" pod "9a14e39a-7816-4289-af4e-a95614befbf8" (UID: "9a14e39a-7816-4289-af4e-a95614befbf8"). InnerVolumeSpecName "kube-api-access-8rpws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:49.312323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.312305 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 01:17:49.409571 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.409533 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z\" (UniqueName: \"kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z\") pod \"66b80d61-39d5-465d-a156-45fcd3ad0722\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " Nov 03 01:17:49.409804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.409785 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume\") pod \"66b80d61-39d5-465d-a156-45fcd3ad0722\" (UID: \"66b80d61-39d5-465d-a156-45fcd3ad0722\") " Nov 03 01:17:49.410133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.410113 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume" pod "66b80d61-39d5-465d-a156-45fcd3ad0722" (UID: "66b80d61-39d5-465d-a156-45fcd3ad0722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:17:49.410242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.410229 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a14e39a-7816-4289-af4e-a95614befbf8-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.410293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.410244 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66b80d61-39d5-465d-a156-45fcd3ad0722-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.410293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.410254 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8rpws\" (UniqueName: \"kubernetes.io/projected/9a14e39a-7816-4289-af4e-a95614befbf8-kube-api-access-8rpws\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.411939 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.411919 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z" pod "66b80d61-39d5-465d-a156-45fcd3ad0722" (UID: "66b80d61-39d5-465d-a156-45fcd3ad0722"). InnerVolumeSpecName "kube-api-access-dr74z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:17:49.511227 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.511189 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr74z\" (UniqueName: \"kubernetes.io/projected/66b80d61-39d5-465d-a156-45fcd3ad0722-kube-api-access-dr74z\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:17:49.920804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.920722 5144 generic.go:356] "Generic (PLEG): container finished" podID="9a14e39a-7816-4289-af4e-a95614befbf8" containerID="0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5" exitCode=0 Nov 03 01:17:49.920804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.920775 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 01:17:49.921020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.920802 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" event={"ID":"9a14e39a-7816-4289-af4e-a95614befbf8","Type":"ContainerDied","Data":"0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5"} Nov 03 01:17:49.921020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.920852 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" event={"ID":"9a14e39a-7816-4289-af4e-a95614befbf8","Type":"ContainerDied","Data":"2981a2b9ed39c609bb797b6add9318bf65d03348f76c1574d33399aa2ba54538"} Nov 03 01:17:49.921020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.920875 5144 scope.go:122] "RemoveContainer" containerID="0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5" Nov 03 01:17:49.922753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.922542 5144 generic.go:356] "Generic (PLEG): container finished" podID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerID="83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972" exitCode=0 Nov 03 01:17:49.922753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.922588 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" Nov 03 01:17:49.922753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.922662 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" event={"ID":"66b80d61-39d5-465d-a156-45fcd3ad0722","Type":"ContainerDied","Data":"83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972"} Nov 03 01:17:49.922753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.922704 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9" event={"ID":"66b80d61-39d5-465d-a156-45fcd3ad0722","Type":"ContainerDied","Data":"aa008ceec0018dd7ec164ec8eb81273a71c295f13fc6773d8e70c2b314e5eb75"} Nov 03 01:17:49.939098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.939075 5144 scope.go:122] "RemoveContainer" containerID="0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5" Nov 03 01:17:49.939415 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:17:49.939395 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5\": container with ID starting with 0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5 not found: ID does not exist" containerID="0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5" Nov 03 01:17:49.939538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.939420 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5"} err="failed to get container status \"0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5\": rpc error: code = NotFound desc = could not find container \"0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5\": container with ID starting with 0a1dbfd09de6deb2a5b8dc5c8bc288599d6954bd5aa2404979b979dc856ce3b5 not found: ID does not exist" Nov 03 01:17:49.939538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.939435 5144 scope.go:122] "RemoveContainer" containerID="83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972" Nov 03 01:17:49.950280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.950265 5144 scope.go:122] "RemoveContainer" containerID="83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972" Nov 03 01:17:49.950577 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:17:49.950560 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972\": container with ID starting with 83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972 not found: ID does not exist" containerID="83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972" Nov 03 01:17:49.950671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:49.950583 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972"} err="failed to get container status \"83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972\": rpc error: code = NotFound desc = could not find container \"83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972\": container with ID starting with 83f2769b0d7a031b3d9bfcaa2e4f92fda38671f48976cb2011fc5729d9a50972 not found: ID does not exist" Nov 03 01:17:50.016872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:50.016833 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.245:50052\" within 1s: context deadline exceeded" Nov 03 01:17:50.017400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:50.016918 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8" Nov 03 01:17:50.084064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:50.084031 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8"] Nov 03 01:17:50.088622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:50.088595 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c87e2f-846976f495-pdzf8"] Nov 03 01:17:50.231741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:50.231618 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9"] Nov 03 01:17:50.235081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:50.235057 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c87e2f-6596d94f95-qf7d9"] Nov 03 01:17:51.996680 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:51.996625 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" path="/var/lib/kubelet/pods/66b80d61-39d5-465d-a156-45fcd3ad0722/volumes" Nov 03 01:17:51.997171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:17:51.997153 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" path="/var/lib/kubelet/pods/9a14e39a-7816-4289-af4e-a95614befbf8/volumes" Nov 03 01:18:07.231110 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.231075 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8"] Nov 03 01:18:07.231523 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.231314 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" containerID="cri-o://6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce" gracePeriod=30 Nov 03 01:18:07.279872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.279837 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp"] Nov 03 01:18:07.280125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.280096 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" containerID="cri-o://a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac" gracePeriod=30 Nov 03 01:18:07.283400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.283374 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56"] Nov 03 01:18:07.283614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.283584 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" containerID="cri-o://919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88" gracePeriod=30 Nov 03 01:18:07.329467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.329433 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p"] Nov 03 01:18:07.329709 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:07.329681 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" containerID="cri-o://0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51" gracePeriod=30 Nov 03 01:18:09.972700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:09.969471 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:09.972700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:09.969532 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:10.417707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.417618 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:10.417991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.417719 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:10.432856 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.432824 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-928s8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:10.432984 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.432909 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:10.554614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.554565 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:18:10.554614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.554643 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:18:10.786126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.786097 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj"] Nov 03 01:18:10.786358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:10.786316 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" containerID="cri-o://f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899" gracePeriod=30 Nov 03 01:18:12.154909 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:12.154874 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:18:12.613065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.612980 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:18:12.635807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.635777 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:18:12.675482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.675459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:18:12.700281 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.700259 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:18:12.800746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800717 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.800746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800753 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.800977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800775 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.800977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800824 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.800977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800852 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.800977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800879 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.800977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800922 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.800977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800954 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.800998 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801028 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801057 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801088 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801159 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801197 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801228 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801258 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.801286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801284 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801291 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801336 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801363 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801403 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801435 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801508 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801548 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801577 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801619 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data\") pod \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\" (UID: \"f1711b6b-c6fc-4afc-aa11-1c14f46017c1\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801670 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801710 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801765 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801796 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801807 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.801842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801829 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801902 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.801935 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802003 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802046 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo\") pod \"265ce86d-3dcc-456b-b248-614f50db0988\" (UID: \"265ce86d-3dcc-456b-b248-614f50db0988\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802073 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket\") pod \"e90f69fe-4d27-49c8-839b-8c363b471683\" (UID: \"e90f69fe-4d27-49c8-839b-8c363b471683\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802110 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo\") pod \"cc02995a-bc90-4b09-8b2a-2d9e69875588\" (UID: \"cc02995a-bc90-4b09-8b2a-2d9e69875588\") " Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802217 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802270 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802822 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.802894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802846 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e90f69fe-4d27-49c8-839b-8c363b471683-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.803462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.802873 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.803462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.803005 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.803462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.803284 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.803668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.803522 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.803775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.803757 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.804192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.804161 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.804548 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.804525 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:12.805349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.804847 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.805349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.804995 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "kube-api-access-bpdsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.805349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.805070 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.805349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.805175 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.805349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.805213 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.805888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.805434 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "kube-api-access-xjlx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.805888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.805670 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.806051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.805928 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:12.806051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.806011 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.806520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.806498 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.806911 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.806528 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.808373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.808349 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.808828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.808802 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:12.808933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.808891 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:12.809005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.808958 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.809424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809273 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:12.809424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809349 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "kube-api-access-cbjzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.809424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809392 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:12.809607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809434 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.809607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809479 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy" pod "f1711b6b-c6fc-4afc-aa11-1c14f46017c1" (UID: "f1711b6b-c6fc-4afc-aa11-1c14f46017c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.809607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809524 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:12.809833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.809743 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.810183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.810159 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy" pod "e90f69fe-4d27-49c8-839b-8c363b471683" (UID: "e90f69fe-4d27-49c8-839b-8c363b471683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.810488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.810462 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:12.810757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.810736 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "kube-api-access-nlcp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.811153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.811130 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token" pod "cc02995a-bc90-4b09-8b2a-2d9e69875588" (UID: "cc02995a-bc90-4b09-8b2a-2d9e69875588"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.811664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.811614 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token" pod "265ce86d-3dcc-456b-b248-614f50db0988" (UID: "265ce86d-3dcc-456b-b248-614f50db0988"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:12.904313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904285 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904312 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904322 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904332 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904340 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904350 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlcp4\" (UniqueName: \"kubernetes.io/projected/cc02995a-bc90-4b09-8b2a-2d9e69875588-kube-api-access-nlcp4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904359 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpdsk\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-kube-api-access-bpdsk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904367 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904376 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904385 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904394 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904402 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904410 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904418 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904426 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904433 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904442 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904450 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904458 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904465 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/265ce86d-3dcc-456b-b248-614f50db0988-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904472 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/265ce86d-3dcc-456b-b248-614f50db0988-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904482 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904490 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e90f69fe-4d27-49c8-839b-8c363b471683-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904498 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc02995a-bc90-4b09-8b2a-2d9e69875588-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904506 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc02995a-bc90-4b09-8b2a-2d9e69875588-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904514 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904522 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/265ce86d-3dcc-456b-b248-614f50db0988-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.904518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904530 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/265ce86d-3dcc-456b-b248-614f50db0988-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.905366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904538 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e90f69fe-4d27-49c8-839b-8c363b471683-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.905366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904545 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc02995a-bc90-4b09-8b2a-2d9e69875588-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.905366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904554 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjzp\" (UniqueName: \"kubernetes.io/projected/e90f69fe-4d27-49c8-839b-8c363b471683-kube-api-access-cbjzp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:12.905366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:12.904562 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjlx8\" (UniqueName: \"kubernetes.io/projected/f1711b6b-c6fc-4afc-aa11-1c14f46017c1-kube-api-access-xjlx8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:13.115798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.115763 5144 generic.go:356] "Generic (PLEG): container finished" podID="265ce86d-3dcc-456b-b248-614f50db0988" containerID="919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88" exitCode=0 Nov 03 01:18:13.115798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.115799 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" Nov 03 01:18:13.116067 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.115825 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" event={"ID":"265ce86d-3dcc-456b-b248-614f50db0988","Type":"ContainerDied","Data":"919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88"} Nov 03 01:18:13.116067 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.115886 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56" event={"ID":"265ce86d-3dcc-456b-b248-614f50db0988","Type":"ContainerDied","Data":"9e49b4622895aa4246b4f614f8b5555e449ef7f034255dc072f291e0a6c5679e"} Nov 03 01:18:13.116067 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.115916 5144 scope.go:122] "RemoveContainer" containerID="919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88" Nov 03 01:18:13.117708 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.117687 5144 generic.go:356] "Generic (PLEG): container finished" podID="e90f69fe-4d27-49c8-839b-8c363b471683" containerID="0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51" exitCode=0 Nov 03 01:18:13.117844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.117745 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" Nov 03 01:18:13.117926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.117743 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" event={"ID":"e90f69fe-4d27-49c8-839b-8c363b471683","Type":"ContainerDied","Data":"0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51"} Nov 03 01:18:13.117983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.117906 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p" event={"ID":"e90f69fe-4d27-49c8-839b-8c363b471683","Type":"ContainerDied","Data":"e87a869e826e9aeee93ae89c62cfe3c0355a75ae21f9620d5e97aa9989414ee9"} Nov 03 01:18:13.119442 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.119417 5144 generic.go:356] "Generic (PLEG): container finished" podID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerID="6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce" exitCode=0 Nov 03 01:18:13.119695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.119676 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" event={"ID":"f1711b6b-c6fc-4afc-aa11-1c14f46017c1","Type":"ContainerDied","Data":"6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce"} Nov 03 01:18:13.119788 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.119722 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" event={"ID":"f1711b6b-c6fc-4afc-aa11-1c14f46017c1","Type":"ContainerDied","Data":"40707da4a1d6193264c8004bc50aa38f03a56152d31e7cda4db93995a9828c32"} Nov 03 01:18:13.119851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.119793 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8" Nov 03 01:18:13.125936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.125911 5144 generic.go:356] "Generic (PLEG): container finished" podID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerID="a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac" exitCode=0 Nov 03 01:18:13.126049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.125973 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" Nov 03 01:18:13.126049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.126035 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" event={"ID":"cc02995a-bc90-4b09-8b2a-2d9e69875588","Type":"ContainerDied","Data":"a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac"} Nov 03 01:18:13.126241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.126075 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp" event={"ID":"cc02995a-bc90-4b09-8b2a-2d9e69875588","Type":"ContainerDied","Data":"9ab5ccc20a9d4625aeaaa95e446034e27ccf7ea4caadde2673c3309acc87f7bd"} Nov 03 01:18:13.142485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.142463 5144 scope.go:122] "RemoveContainer" containerID="919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88" Nov 03 01:18:13.143601 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:13.143576 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88\": container with ID starting with 919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88 not found: ID does not exist" containerID="919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88" Nov 03 01:18:13.144053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.143611 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88"} err="failed to get container status \"919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88\": rpc error: code = NotFound desc = could not find container \"919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88\": container with ID starting with 919d323c4420781f718e35667dff7bac63c77e6ad8a2d394a55781c97c434d88 not found: ID does not exist" Nov 03 01:18:13.144053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.144017 5144 scope.go:122] "RemoveContainer" containerID="0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51" Nov 03 01:18:13.159665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.159647 5144 scope.go:122] "RemoveContainer" containerID="0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51" Nov 03 01:18:13.160032 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:13.160003 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51\": container with ID starting with 0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51 not found: ID does not exist" containerID="0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51" Nov 03 01:18:13.160081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.160043 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51"} err="failed to get container status \"0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51\": rpc error: code = NotFound desc = could not find container \"0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51\": container with ID starting with 0d94f1deaea11bf489f323e021fcaa307a152999bba14c6bc33345fa06287f51 not found: ID does not exist" Nov 03 01:18:13.160081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.160068 5144 scope.go:122] "RemoveContainer" containerID="6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce" Nov 03 01:18:13.171102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.171078 5144 scope.go:122] "RemoveContainer" containerID="6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce" Nov 03 01:18:13.171440 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:13.171409 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce\": container with ID starting with 6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce not found: ID does not exist" containerID="6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce" Nov 03 01:18:13.171556 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.171448 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce"} err="failed to get container status \"6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce\": rpc error: code = NotFound desc = could not find container \"6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce\": container with ID starting with 6f071a235129e7a00cdc09c3c9c2c99bbbbe8ec08b34ed7ffa096e2b028080ce not found: ID does not exist" Nov 03 01:18:13.171556 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.171471 5144 scope.go:122] "RemoveContainer" containerID="a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac" Nov 03 01:18:13.181266 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.181248 5144 scope.go:122] "RemoveContainer" containerID="a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac" Nov 03 01:18:13.181604 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:13.181582 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac\": container with ID starting with a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac not found: ID does not exist" containerID="a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac" Nov 03 01:18:13.181687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.181614 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac"} err="failed to get container status \"a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac\": rpc error: code = NotFound desc = could not find container \"a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac\": container with ID starting with a163a4229daf3b4ca7ede15914c87da394bf662eb9ab2a155e7281f43fe5c6ac not found: ID does not exist" Nov 03 01:18:13.303801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.303774 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56"] Nov 03 01:18:13.308199 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.308171 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-9mv56"] Nov 03 01:18:13.520690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.520663 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p"] Nov 03 01:18:13.524522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.524499 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6ck2p"] Nov 03 01:18:13.632307 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.632271 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8"] Nov 03 01:18:13.639706 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.639681 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-928s8"] Nov 03 01:18:13.750473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.750437 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp"] Nov 03 01:18:13.759563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.759537 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-d8bkp"] Nov 03 01:18:13.901262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.901191 5144 scope.go:122] "RemoveContainer" containerID="79dd4d79c7cb17e8f58c871940aa6a0a963550497cd5cfd91293ef127b7ad559" Nov 03 01:18:13.996354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.996328 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="265ce86d-3dcc-456b-b248-614f50db0988" path="/var/lib/kubelet/pods/265ce86d-3dcc-456b-b248-614f50db0988/volumes" Nov 03 01:18:13.996789 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.996776 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" path="/var/lib/kubelet/pods/cc02995a-bc90-4b09-8b2a-2d9e69875588/volumes" Nov 03 01:18:13.997178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.997166 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" path="/var/lib/kubelet/pods/e90f69fe-4d27-49c8-839b-8c363b471683/volumes" Nov 03 01:18:13.997838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:13.997824 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" path="/var/lib/kubelet/pods/f1711b6b-c6fc-4afc-aa11-1c14f46017c1/volumes" Nov 03 01:18:16.065147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.065089 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:18:16.138023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.137995 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138050 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138125 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138173 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138322 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138299 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138365 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:16.138408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138381 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138420 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:16.138520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138431 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138498 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138715 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138548 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo\") pod \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\" (UID: \"d8b57094-a4ae-470d-8842-197f9bd4e7c6\") " Nov 03 01:18:16.138715 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138601 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:16.138827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.138790 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:16.139165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.139087 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.139165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.139111 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.139165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.139126 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.139165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.139139 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.139497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.139461 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:16.140845 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.140816 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:16.141026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.141001 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:16.141717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.141697 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:18:16.141824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.141736 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk" pod "d8b57094-a4ae-470d-8842-197f9bd4e7c6" (UID: "d8b57094-a4ae-470d-8842-197f9bd4e7c6"). InnerVolumeSpecName "kube-api-access-49pbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:16.155742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.155710 5144 generic.go:356] "Generic (PLEG): container finished" podID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerID="f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899" exitCode=0 Nov 03 01:18:16.155858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.155771 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" event={"ID":"d8b57094-a4ae-470d-8842-197f9bd4e7c6","Type":"ContainerDied","Data":"f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899"} Nov 03 01:18:16.155858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.155783 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" Nov 03 01:18:16.155858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.155825 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj" event={"ID":"d8b57094-a4ae-470d-8842-197f9bd4e7c6","Type":"ContainerDied","Data":"edf4af348460a5dfad07d09e488d9699f48c4e8c248dbb974c48f0450523d1e4"} Nov 03 01:18:16.155858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.155855 5144 scope.go:122] "RemoveContainer" containerID="f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899" Nov 03 01:18:16.169730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.169712 5144 scope.go:122] "RemoveContainer" containerID="547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3" Nov 03 01:18:16.179034 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.179019 5144 scope.go:122] "RemoveContainer" containerID="f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899" Nov 03 01:18:16.179279 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:16.179257 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899\": container with ID starting with f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899 not found: ID does not exist" containerID="f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899" Nov 03 01:18:16.179395 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.179286 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899"} err="failed to get container status \"f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899\": rpc error: code = NotFound desc = could not find container \"f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899\": container with ID starting with f500fb17c192e311731219972de3e01f01575ebde73841a7357b4bbb9b3d6899 not found: ID does not exist" Nov 03 01:18:16.179395 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.179307 5144 scope.go:122] "RemoveContainer" containerID="547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3" Nov 03 01:18:16.179609 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:16.179585 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3\": container with ID starting with 547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3 not found: ID does not exist" containerID="547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3" Nov 03 01:18:16.179748 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.179615 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3"} err="failed to get container status \"547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3\": rpc error: code = NotFound desc = could not find container \"547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3\": container with ID starting with 547cb1bf52a39912ff504a23719677ea9954ee641c65e6786cf9ebeffc8b7aa3 not found: ID does not exist" Nov 03 01:18:16.240721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.240695 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.240721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.240718 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.240942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.240730 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.240942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.240749 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49pbk\" (UniqueName: \"kubernetes.io/projected/d8b57094-a4ae-470d-8842-197f9bd4e7c6-kube-api-access-49pbk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.240942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.240759 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d8b57094-a4ae-470d-8842-197f9bd4e7c6-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:16.345205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.342455 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj"] Nov 03 01:18:16.346048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:16.346024 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-aee0dd37-openshift-default-767bc57fc-9wjgj"] Nov 03 01:18:18.004112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:18.004075 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" path="/var/lib/kubelet/pods/d8b57094-a4ae-470d-8842-197f9bd4e7c6/volumes" Nov 03 01:18:32.655583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.655547 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt"] Nov 03 01:18:32.656385 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656369 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" Nov 03 01:18:32.656441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656388 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" Nov 03 01:18:32.656441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656402 5144 cpu_manager.go:414] "containerMap: removing container" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" Nov 03 01:18:32.656441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656409 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" Nov 03 01:18:32.656441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656434 5144 cpu_manager.go:414] "containerMap: removing container" podUID="83a0c076-8805-4771-8e54-99f1333aff51" containerName="payload-pre-processing" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656442 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a0c076-8805-4771-8e54-99f1333aff51" containerName="payload-pre-processing" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656463 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656470 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656481 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656489 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656514 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c76fc554-99de-4249-aed2-5e9d99fe9636" containerName="payload-processing" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656520 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c76fc554-99de-4249-aed2-5e9d99fe9636" containerName="payload-processing" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656544 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" Nov 03 01:18:32.656567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656550 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656570 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7de899b5-3ced-4c50-9a52-595ef4764915" containerName="revoke-keys" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656577 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7de899b5-3ced-4c50-9a52-595ef4764915" containerName="revoke-keys" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656591 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656597 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656608 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656614 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656642 5144 cpu_manager.go:414] "containerMap: removing container" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerName="payload-pre-processing" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656650 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerName="payload-pre-processing" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656770 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7de899b5-3ced-4c50-9a52-595ef4764915" containerName="revoke-keys" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656785 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90f69fe-4d27-49c8-839b-8c363b471683" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656793 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd6edfa-28b9-4c2d-8983-377ec06c0e29" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656801 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="66b80d61-39d5-465d-a156-45fcd3ad0722" containerName="payload-pre-processing" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656809 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a14e39a-7816-4289-af4e-a95614befbf8" containerName="payload-processing" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656817 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="265ce86d-3dcc-456b-b248-614f50db0988" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656828 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a0c076-8805-4771-8e54-99f1333aff51" containerName="payload-pre-processing" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656838 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc02995a-bc90-4b09-8b2a-2d9e69875588" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656846 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656856 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c76fc554-99de-4249-aed2-5e9d99fe9636" containerName="payload-processing" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656864 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" Nov 03 01:18:32.656880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.656875 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1711b6b-c6fc-4afc-aa11-1c14f46017c1" containerName="istio-proxy" Nov 03 01:18:32.661155 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.661133 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.714264 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.714237 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt"] Nov 03 01:18:32.758512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.758486 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:32.758711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.758588 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:18:32.820613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.820580 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4q7\" (UniqueName: \"kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.820782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.820659 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.820782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.820722 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.922046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.921961 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4q7\" (UniqueName: \"kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.922046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.922006 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.922251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.922049 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.923021 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.922980 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.924797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.924765 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:32.980114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:32.980088 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4q7\" (UniqueName: \"kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7\") pod \"maas-api-revoke-sub-6f28d171-999gt\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:33.276133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:33.276096 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:33.652786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:33.652760 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt"] Nov 03 01:18:33.654300 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:18:33.654269 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823add80_767e_465c_abbc_936177af2629.slice/crio-c1f192b10955d99ff79f1f2ef2d6701def4a6482df1557c6293a5b05c139dac3 WatchSource:0}: Error finding container c1f192b10955d99ff79f1f2ef2d6701def4a6482df1557c6293a5b05c139dac3: Status 404 returned error can't find the container with id c1f192b10955d99ff79f1f2ef2d6701def4a6482df1557c6293a5b05c139dac3 Nov 03 01:18:34.290912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:34.290811 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" event={"ID":"823add80-767e-465c-abbc-936177af2629","Type":"ContainerStarted","Data":"f005538ee76375fdd5d49886ce3759623b6b2e90c44092fd11b64d0cbc6834cb"} Nov 03 01:18:34.291387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:34.290924 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" event={"ID":"823add80-767e-465c-abbc-936177af2629","Type":"ContainerStarted","Data":"c1f192b10955d99ff79f1f2ef2d6701def4a6482df1557c6293a5b05c139dac3"} Nov 03 01:18:34.404639 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:34.404583 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" podStartSLOduration=2.404568863 podStartE2EDuration="2.404568863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:34.402839213 +0000 UTC m=+4404.548533690" watchObservedRunningTime="2026-11-03 01:18:34.404568863 +0000 UTC m=+4404.550263318" Nov 03 01:18:35.299657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:35.299602 5144 generic.go:356] "Generic (PLEG): container finished" podID="823add80-767e-465c-abbc-936177af2629" containerID="f005538ee76375fdd5d49886ce3759623b6b2e90c44092fd11b64d0cbc6834cb" exitCode=0 Nov 03 01:18:35.300097 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:35.299687 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" event={"ID":"823add80-767e-465c-abbc-936177af2629","Type":"ContainerDied","Data":"f005538ee76375fdd5d49886ce3759623b6b2e90c44092fd11b64d0cbc6834cb"} Nov 03 01:18:36.468798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.468774 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:36.564832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.564762 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca\") pod \"823add80-767e-465c-abbc-936177af2629\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " Nov 03 01:18:36.564832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.564812 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7\" (UniqueName: \"kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7\") pod \"823add80-767e-465c-abbc-936177af2629\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " Nov 03 01:18:36.565011 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.564972 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca\") pod \"823add80-767e-465c-abbc-936177af2629\" (UID: \"823add80-767e-465c-abbc-936177af2629\") " Nov 03 01:18:36.565106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.565085 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca" pod "823add80-767e-465c-abbc-936177af2629" (UID: "823add80-767e-465c-abbc-936177af2629"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:36.565382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.565361 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/823add80-767e-465c-abbc-936177af2629-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:36.567174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.567152 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7" pod "823add80-767e-465c-abbc-936177af2629" (UID: "823add80-767e-465c-abbc-936177af2629"). InnerVolumeSpecName "kube-api-access-wz4q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:36.567271 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.567152 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca" pod "823add80-767e-465c-abbc-936177af2629" (UID: "823add80-767e-465c-abbc-936177af2629"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:36.666561 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.666518 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4q7\" (UniqueName: \"kubernetes.io/projected/823add80-767e-465c-abbc-936177af2629-kube-api-access-wz4q7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:36.666561 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:36.666558 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/823add80-767e-465c-abbc-936177af2629-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:37.018300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:37.018268 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d"] Nov 03 01:18:37.316187 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:37.316106 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" Nov 03 01:18:37.316187 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:37.316139 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt" event={"ID":"823add80-767e-465c-abbc-936177af2629","Type":"ContainerDied","Data":"c1f192b10955d99ff79f1f2ef2d6701def4a6482df1557c6293a5b05c139dac3"} Nov 03 01:18:37.316187 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:37.316186 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1f192b10955d99ff79f1f2ef2d6701def4a6482df1557c6293a5b05c139dac3" Nov 03 01:18:42.157787 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:42.157749 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:18:45.408506 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.408477 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd"] Nov 03 01:18:45.409116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.409099 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" Nov 03 01:18:45.409116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.409117 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8b57094-a4ae-470d-8842-197f9bd4e7c6" containerName="istio-proxy" Nov 03 01:18:45.409287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.409142 5144 cpu_manager.go:414] "containerMap: removing container" podUID="823add80-767e-465c-abbc-936177af2629" containerName="revoke-keys" Nov 03 01:18:45.409287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.409148 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="823add80-767e-465c-abbc-936177af2629" containerName="revoke-keys" Nov 03 01:18:45.409287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.409240 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="823add80-767e-465c-abbc-936177af2629" containerName="revoke-keys" Nov 03 01:18:45.413106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.413086 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.462969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.462941 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.463177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.462975 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.463177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.462994 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhv4x\" (UniqueName: \"kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.468513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.468483 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd"] Nov 03 01:18:45.513922 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.513882 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:18:45.514112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.513929 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:18:45.566728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.566376 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.566728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.566447 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.566728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.566479 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhv4x\" (UniqueName: \"kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.568187 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.568155 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.571084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.571062 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.599304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.599275 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7"] Nov 03 01:18:45.623289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.623254 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhv4x\" (UniqueName: \"kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x\") pod \"maas-api-revoke-sub-7a79ba91-qzrpd\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:45.724751 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.724653 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt"] Nov 03 01:18:45.732117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:45.732090 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:46.115305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:46.115276 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd"] Nov 03 01:18:46.115445 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:18:46.115421 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd68a66_a0b8_4fdc_b63a_45ac2597deee.slice/crio-b78be8f9abae56bf55c7c72e3b5fee580a33fca639b9347acb3131c6cf9ef125 WatchSource:0}: Error finding container b78be8f9abae56bf55c7c72e3b5fee580a33fca639b9347acb3131c6cf9ef125: Status 404 returned error can't find the container with id b78be8f9abae56bf55c7c72e3b5fee580a33fca639b9347acb3131c6cf9ef125 Nov 03 01:18:46.387836 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:46.387764 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" event={"ID":"ffd68a66-a0b8-4fdc-b63a-45ac2597deee","Type":"ContainerStarted","Data":"b78be8f9abae56bf55c7c72e3b5fee580a33fca639b9347acb3131c6cf9ef125"} Nov 03 01:18:47.396774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:47.396735 5144 generic.go:356] "Generic (PLEG): container finished" podID="ffd68a66-a0b8-4fdc-b63a-45ac2597deee" containerID="dc8f3c4dd456a723086e5551692a185be4146a0d66962940b10d7194a30fbf21" exitCode=0 Nov 03 01:18:47.397169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:47.396814 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" event={"ID":"ffd68a66-a0b8-4fdc-b63a-45ac2597deee","Type":"ContainerDied","Data":"dc8f3c4dd456a723086e5551692a185be4146a0d66962940b10d7194a30fbf21"} Nov 03 01:18:48.597365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.597340 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:48.699494 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.699429 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca\") pod \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " Nov 03 01:18:48.699657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.699538 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x\" (UniqueName: \"kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x\") pod \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " Nov 03 01:18:48.699657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.699564 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca\") pod \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\" (UID: \"ffd68a66-a0b8-4fdc-b63a-45ac2597deee\") " Nov 03 01:18:48.699965 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.699941 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca" pod "ffd68a66-a0b8-4fdc-b63a-45ac2597deee" (UID: "ffd68a66-a0b8-4fdc-b63a-45ac2597deee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:18:48.700233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.700029 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:48.701553 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.701531 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca" pod "ffd68a66-a0b8-4fdc-b63a-45ac2597deee" (UID: "ffd68a66-a0b8-4fdc-b63a-45ac2597deee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:48.701798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.701784 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x" pod "ffd68a66-a0b8-4fdc-b63a-45ac2597deee" (UID: "ffd68a66-a0b8-4fdc-b63a-45ac2597deee"). InnerVolumeSpecName "kube-api-access-fhv4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:48.801462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.801430 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhv4x\" (UniqueName: \"kubernetes.io/projected/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-kube-api-access-fhv4x\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:48.801462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:48.801461 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd68a66-a0b8-4fdc-b63a-45ac2597deee-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:49.417735 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:49.417703 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" Nov 03 01:18:49.417735 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:49.417722 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd" event={"ID":"ffd68a66-a0b8-4fdc-b63a-45ac2597deee","Type":"ContainerDied","Data":"b78be8f9abae56bf55c7c72e3b5fee580a33fca639b9347acb3131c6cf9ef125"} Nov 03 01:18:49.417955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:49.417769 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b78be8f9abae56bf55c7c72e3b5fee580a33fca639b9347acb3131c6cf9ef125" Nov 03 01:18:50.599671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:50.599576 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerName="payload-processing" containerID="cri-o://91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00" gracePeriod=25 Nov 03 01:18:50.725894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:50.725840 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerName="payload-pre-processing" containerID="cri-o://c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4" gracePeriod=25 Nov 03 01:18:52.039926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.039886 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" containerID="cri-o://c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34" gracePeriod=45 Nov 03 01:18:52.298925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.298869 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 01:18:52.341904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.341874 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache\") pod \"5a5a804f-2741-40ac-81e3-372308a66d5e\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " Nov 03 01:18:52.342052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.341929 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm\") pod \"5a5a804f-2741-40ac-81e3-372308a66d5e\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " Nov 03 01:18:52.342052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.341975 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs\") pod \"5a5a804f-2741-40ac-81e3-372308a66d5e\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " Nov 03 01:18:52.342052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342036 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c\" (UniqueName: \"kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c\") pod \"5a5a804f-2741-40ac-81e3-372308a66d5e\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " Nov 03 01:18:52.342234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342153 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir\") pod \"5a5a804f-2741-40ac-81e3-372308a66d5e\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " Nov 03 01:18:52.342234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342159 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache" pod "5a5a804f-2741-40ac-81e3-372308a66d5e" (UID: "5a5a804f-2741-40ac-81e3-372308a66d5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.342234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342203 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home\") pod \"5a5a804f-2741-40ac-81e3-372308a66d5e\" (UID: \"5a5a804f-2741-40ac-81e3-372308a66d5e\") " Nov 03 01:18:52.342511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342482 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir" pod "5a5a804f-2741-40ac-81e3-372308a66d5e" (UID: "5a5a804f-2741-40ac-81e3-372308a66d5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.342511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342495 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home" pod "5a5a804f-2741-40ac-81e3-372308a66d5e" (UID: "5a5a804f-2741-40ac-81e3-372308a66d5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.342890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342869 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.342944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342896 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.342944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.342913 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.344317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.344297 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm" pod "5a5a804f-2741-40ac-81e3-372308a66d5e" (UID: "5a5a804f-2741-40ac-81e3-372308a66d5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:18:52.344774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.344755 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs" pod "5a5a804f-2741-40ac-81e3-372308a66d5e" (UID: "5a5a804f-2741-40ac-81e3-372308a66d5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:18:52.344842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.344785 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c" pod "5a5a804f-2741-40ac-81e3-372308a66d5e" (UID: "5a5a804f-2741-40ac-81e3-372308a66d5e"). InnerVolumeSpecName "kube-api-access-wv28c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:18:52.443134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443093 5144 generic.go:356] "Generic (PLEG): container finished" podID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerID="c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34" exitCode=0 Nov 03 01:18:52.443338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443318 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" event={"ID":"5a5a804f-2741-40ac-81e3-372308a66d5e","Type":"ContainerDied","Data":"c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34"} Nov 03 01:18:52.443406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443363 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" event={"ID":"5a5a804f-2741-40ac-81e3-372308a66d5e","Type":"ContainerDied","Data":"7369c9c5d8532cb22a5cfb1588addbd9e6fe8af59752d791c4daacb1d7949001"} Nov 03 01:18:52.443406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443387 5144 scope.go:122] "RemoveContainer" containerID="c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34" Nov 03 01:18:52.443582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443568 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d" Nov 03 01:18:52.443831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443807 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5a5a804f-2741-40ac-81e3-372308a66d5e-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.444027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.443982 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5a5a804f-2741-40ac-81e3-372308a66d5e-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.444027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.444009 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv28c\" (UniqueName: \"kubernetes.io/projected/5a5a804f-2741-40ac-81e3-372308a66d5e-kube-api-access-wv28c\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:18:52.458150 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.458129 5144 scope.go:122] "RemoveContainer" containerID="c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34" Nov 03 01:18:52.458565 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:18:52.458541 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34\": container with ID starting with c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34 not found: ID does not exist" containerID="c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34" Nov 03 01:18:52.458682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.458575 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34"} err="failed to get container status \"c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34\": rpc error: code = NotFound desc = could not find container \"c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34\": container with ID starting with c69e8342bf17af150fc687d3135d10438f469510f697e546847d442ac30fcc34 not found: ID does not exist" Nov 03 01:18:52.607879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.607789 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d"] Nov 03 01:18:52.610929 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:52.610904 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-d7bfc2/premium-w3-d7bfc2-kserve-64bd465479-b562d"] Nov 03 01:18:53.996522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:53.996489 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" path="/var/lib/kubelet/pods/5a5a804f-2741-40ac-81e3-372308a66d5e/volumes" Nov 03 01:18:54.683784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:54.683747 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg"] Nov 03 01:18:54.789431 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:54.789403 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7"] Nov 03 01:18:59.487506 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:59.487461 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:18:59.488004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:59.487540 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:18:59.684532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:59.684497 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerName="payload-processing" containerID="cri-o://54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e" gracePeriod=25 Nov 03 01:18:59.790124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:18:59.790089 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerName="payload-pre-processing" containerID="cri-o://03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3" gracePeriod=25 Nov 03 01:19:04.764182 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:04.764138 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:19:04.764740 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:04.764277 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:19:05.017899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.017815 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn"] Nov 03 01:19:05.018692 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.018670 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ffd68a66-a0b8-4fdc-b63a-45ac2597deee" containerName="revoke-keys" Nov 03 01:19:05.018692 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.018692 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd68a66-a0b8-4fdc-b63a-45ac2597deee" containerName="revoke-keys" Nov 03 01:19:05.018880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.018722 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" Nov 03 01:19:05.018880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.018730 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" Nov 03 01:19:05.018980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.018907 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd68a66-a0b8-4fdc-b63a-45ac2597deee" containerName="revoke-keys" Nov 03 01:19:05.018980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.018921 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a5a804f-2741-40ac-81e3-372308a66d5e" containerName="main" Nov 03 01:19:05.023597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.023571 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.077995 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.077954 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn"] Nov 03 01:19:05.092316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.092285 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.092457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.092337 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.092539 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.092462 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxzzs\" (UniqueName: \"kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.122764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.122737 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:05.122872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.122782 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:19:05.194050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.194023 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.194177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.194060 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.194177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.194090 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxzzs\" (UniqueName: \"kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.194685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.194666 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.196537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.196519 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.250648 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.250607 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxzzs\" (UniqueName: \"kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs\") pod \"maas-api-revoke-sub-c162b859-qtrcn\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.337287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.337230 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:05.711259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:05.711216 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn"] Nov 03 01:19:05.712517 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:05.712488 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dfe161_0d32_467c_8fda_8aa78362b33f.slice/crio-85d01312779862112a6e4a785337c220166fca496249cc424dd3285eaddceb1d WatchSource:0}: Error finding container 85d01312779862112a6e4a785337c220166fca496249cc424dd3285eaddceb1d: Status 404 returned error can't find the container with id 85d01312779862112a6e4a785337c220166fca496249cc424dd3285eaddceb1d Nov 03 01:19:06.553006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:06.552972 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" event={"ID":"f5dfe161-0d32-467c-8fda-8aa78362b33f","Type":"ContainerStarted","Data":"4164c20a887b2d27a0f4335e2f6d158ef73bcb21c0fd1adc64c56b673b7756f6"} Nov 03 01:19:06.553450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:06.553052 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" event={"ID":"f5dfe161-0d32-467c-8fda-8aa78362b33f","Type":"ContainerStarted","Data":"85d01312779862112a6e4a785337c220166fca496249cc424dd3285eaddceb1d"} Nov 03 01:19:06.667790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:06.667734 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" podStartSLOduration=2.667720362 podStartE2EDuration="2.667720362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:06.666191351 +0000 UTC m=+4436.811885801" watchObservedRunningTime="2026-11-03 01:19:06.667720362 +0000 UTC m=+4436.813414845" Nov 03 01:19:07.560928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:07.560892 5144 generic.go:356] "Generic (PLEG): container finished" podID="f5dfe161-0d32-467c-8fda-8aa78362b33f" containerID="4164c20a887b2d27a0f4335e2f6d158ef73bcb21c0fd1adc64c56b673b7756f6" exitCode=0 Nov 03 01:19:07.561338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:07.560971 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" event={"ID":"f5dfe161-0d32-467c-8fda-8aa78362b33f","Type":"ContainerDied","Data":"4164c20a887b2d27a0f4335e2f6d158ef73bcb21c0fd1adc64c56b673b7756f6"} Nov 03 01:19:08.728357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.728333 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:08.832211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.832133 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca\") pod \"f5dfe161-0d32-467c-8fda-8aa78362b33f\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " Nov 03 01:19:08.832211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.832200 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs\" (UniqueName: \"kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs\") pod \"f5dfe161-0d32-467c-8fda-8aa78362b33f\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " Nov 03 01:19:08.832448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.832247 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca\") pod \"f5dfe161-0d32-467c-8fda-8aa78362b33f\" (UID: \"f5dfe161-0d32-467c-8fda-8aa78362b33f\") " Nov 03 01:19:08.832589 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.832561 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca" pod "f5dfe161-0d32-467c-8fda-8aa78362b33f" (UID: "f5dfe161-0d32-467c-8fda-8aa78362b33f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:08.834283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.834262 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca" pod "f5dfe161-0d32-467c-8fda-8aa78362b33f" (UID: "f5dfe161-0d32-467c-8fda-8aa78362b33f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:08.834457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.834435 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs" pod "f5dfe161-0d32-467c-8fda-8aa78362b33f" (UID: "f5dfe161-0d32-467c-8fda-8aa78362b33f"). InnerVolumeSpecName "kube-api-access-pxzzs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:08.933673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.933623 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxzzs\" (UniqueName: \"kubernetes.io/projected/f5dfe161-0d32-467c-8fda-8aa78362b33f-kube-api-access-pxzzs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:08.933673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.933670 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:08.933892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:08.933681 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dfe161-0d32-467c-8fda-8aa78362b33f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:09.487315 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:09.487270 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:19:09.487513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:09.487313 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:19:09.575949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:09.575912 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" Nov 03 01:19:09.575949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:09.575923 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn" event={"ID":"f5dfe161-0d32-467c-8fda-8aa78362b33f","Type":"ContainerDied","Data":"85d01312779862112a6e4a785337c220166fca496249cc424dd3285eaddceb1d"} Nov 03 01:19:09.576184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:09.575975 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85d01312779862112a6e4a785337c220166fca496249cc424dd3285eaddceb1d" Nov 03 01:19:10.884862 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:10.884835 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 01:19:10.986859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:10.986836 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 01:19:11.052521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.052494 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume\") pod \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " Nov 03 01:19:11.052521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.052528 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p\" (UniqueName: \"kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p\") pod \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\" (UID: \"0a6d2b26-8aa7-4697-ba17-7c537dee13cc\") " Nov 03 01:19:11.052937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.052906 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume" pod "0a6d2b26-8aa7-4697-ba17-7c537dee13cc" (UID: "0a6d2b26-8aa7-4697-ba17-7c537dee13cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:11.054891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.054866 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p" pod "0a6d2b26-8aa7-4697-ba17-7c537dee13cc" (UID: "0a6d2b26-8aa7-4697-ba17-7c537dee13cc"). InnerVolumeSpecName "kube-api-access-zj29p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:11.153302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.153240 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume\") pod \"aaabe548-30c0-4efa-974d-400e2d40d98f\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " Nov 03 01:19:11.153450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.153356 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7\" (UniqueName: \"kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7\") pod \"aaabe548-30c0-4efa-974d-400e2d40d98f\" (UID: \"aaabe548-30c0-4efa-974d-400e2d40d98f\") " Nov 03 01:19:11.153582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.153569 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:11.153623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.153584 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj29p\" (UniqueName: \"kubernetes.io/projected/0a6d2b26-8aa7-4697-ba17-7c537dee13cc-kube-api-access-zj29p\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:11.153701 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.153675 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume" pod "aaabe548-30c0-4efa-974d-400e2d40d98f" (UID: "aaabe548-30c0-4efa-974d-400e2d40d98f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:11.155476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.155457 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7" pod "aaabe548-30c0-4efa-974d-400e2d40d98f" (UID: "aaabe548-30c0-4efa-974d-400e2d40d98f"). InnerVolumeSpecName "kube-api-access-7m8q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:11.255205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.255172 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaabe548-30c0-4efa-974d-400e2d40d98f-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:11.255205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.255198 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m8q7\" (UniqueName: \"kubernetes.io/projected/aaabe548-30c0-4efa-974d-400e2d40d98f-kube-api-access-7m8q7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:11.591835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.591796 5144 generic.go:356] "Generic (PLEG): container finished" podID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerID="c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4" exitCode=0 Nov 03 01:19:11.592047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.591845 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" event={"ID":"aaabe548-30c0-4efa-974d-400e2d40d98f","Type":"ContainerDied","Data":"c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4"} Nov 03 01:19:11.592047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.591883 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" Nov 03 01:19:11.592047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.591942 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt" event={"ID":"aaabe548-30c0-4efa-974d-400e2d40d98f","Type":"ContainerDied","Data":"261ebdea25c6d63c9a778756962c3c0e14271c1daae92ff14cda4883230c1f62"} Nov 03 01:19:11.592047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.591973 5144 scope.go:122] "RemoveContainer" containerID="c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4" Nov 03 01:19:11.593576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.593549 5144 generic.go:356] "Generic (PLEG): container finished" podID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerID="91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00" exitCode=0 Nov 03 01:19:11.593899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.593610 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" Nov 03 01:19:11.593899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.593606 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" event={"ID":"0a6d2b26-8aa7-4697-ba17-7c537dee13cc","Type":"ContainerDied","Data":"91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00"} Nov 03 01:19:11.593899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.593771 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7" event={"ID":"0a6d2b26-8aa7-4697-ba17-7c537dee13cc","Type":"ContainerDied","Data":"58a8e2bd37c73aec9f572dcab6abee88214f536fe6e59520276eb5e15edf86ed"} Nov 03 01:19:11.603653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.603619 5144 scope.go:122] "RemoveContainer" containerID="c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4" Nov 03 01:19:11.603985 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:11.603960 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4\": container with ID starting with c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4 not found: ID does not exist" containerID="c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4" Nov 03 01:19:11.604103 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.603994 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4"} err="failed to get container status \"c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4\": rpc error: code = NotFound desc = could not find container \"c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4\": container with ID starting with c3672f1dc5990ca944f80f6e2f3904e09fc91ebd6b903f3bcd4ff90406be0cc4 not found: ID does not exist" Nov 03 01:19:11.604103 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.604016 5144 scope.go:122] "RemoveContainer" containerID="91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00" Nov 03 01:19:11.623888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.623870 5144 scope.go:122] "RemoveContainer" containerID="91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00" Nov 03 01:19:11.624215 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:11.624196 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00\": container with ID starting with 91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00 not found: ID does not exist" containerID="91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00" Nov 03 01:19:11.624292 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.624221 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00"} err="failed to get container status \"91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00\": rpc error: code = NotFound desc = could not find container \"91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00\": container with ID starting with 91d075b026ef3da766abe8d32f21ef13282fb6627ad3de94ff07d4b8e0f4ba00 not found: ID does not exist" Nov 03 01:19:11.756837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.756809 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7"] Nov 03 01:19:11.806938 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.806911 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-d7bfc2-597494665-rbxb7"] Nov 03 01:19:11.912769 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.912698 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt"] Nov 03 01:19:11.915932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.915913 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-d7bfc2-8d9c46995-7q6lt"] Nov 03 01:19:11.996108 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.996074 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" path="/var/lib/kubelet/pods/0a6d2b26-8aa7-4697-ba17-7c537dee13cc/volumes" Nov 03 01:19:11.996526 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:11.996508 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" path="/var/lib/kubelet/pods/aaabe548-30c0-4efa-974d-400e2d40d98f/volumes" Nov 03 01:19:12.154650 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:12.154607 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:19:13.969957 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:13.969926 5144 scope.go:122] "RemoveContainer" containerID="a60491f963258f21b51581de13674b92ab82984f8d321dcc2d59b94fef1d0cab" Nov 03 01:19:14.764242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:14.764205 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:19:14.764497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:14.764465 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:19:15.764722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.764691 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9"] Nov 03 01:19:15.765516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765498 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerName="payload-processing" Nov 03 01:19:15.765654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765517 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerName="payload-processing" Nov 03 01:19:15.765654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765556 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f5dfe161-0d32-467c-8fda-8aa78362b33f" containerName="revoke-keys" Nov 03 01:19:15.765654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765566 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dfe161-0d32-467c-8fda-8aa78362b33f" containerName="revoke-keys" Nov 03 01:19:15.765654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765600 5144 cpu_manager.go:414] "containerMap: removing container" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerName="payload-pre-processing" Nov 03 01:19:15.765654 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765607 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerName="payload-pre-processing" Nov 03 01:19:15.765840 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765755 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaabe548-30c0-4efa-974d-400e2d40d98f" containerName="payload-pre-processing" Nov 03 01:19:15.765840 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765771 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a6d2b26-8aa7-4697-ba17-7c537dee13cc" containerName="payload-processing" Nov 03 01:19:15.765840 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.765785 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dfe161-0d32-467c-8fda-8aa78362b33f" containerName="revoke-keys" Nov 03 01:19:15.771376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.771355 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:15.823648 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.823612 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9"] Nov 03 01:19:15.867930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.867906 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:15.868055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.867953 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:19:15.904504 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.904480 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:15.904657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.904605 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:15.904714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:15.904673 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c5df\" (UniqueName: \"kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.005666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.005622 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.005832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.005770 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.005832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.005804 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c5df\" (UniqueName: \"kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.006341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.006315 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.008217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.008190 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.063228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.063142 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c5df\" (UniqueName: \"kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df\") pod \"maas-api-revoke-sub-03f92fff-x8rx9\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.084778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.084753 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:16.468218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.468188 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9"] Nov 03 01:19:16.469478 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:16.469445 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3941e883_dd6c_40e4_911a_8c6f46455248.slice/crio-7d2fabb35ee9eff825dc49cefdf5a90f7464514478d5003c7150f7f1c7d985e4 WatchSource:0}: Error finding container 7d2fabb35ee9eff825dc49cefdf5a90f7464514478d5003c7150f7f1c7d985e4: Status 404 returned error can't find the container with id 7d2fabb35ee9eff825dc49cefdf5a90f7464514478d5003c7150f7f1c7d985e4 Nov 03 01:19:16.633973 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:16.633943 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" event={"ID":"3941e883-dd6c-40e4-911a-8c6f46455248","Type":"ContainerStarted","Data":"7d2fabb35ee9eff825dc49cefdf5a90f7464514478d5003c7150f7f1c7d985e4"} Nov 03 01:19:17.642465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:17.642356 5144 generic.go:356] "Generic (PLEG): container finished" podID="3941e883-dd6c-40e4-911a-8c6f46455248" containerID="dceca7f152f8c15bd16dab37ff5b8b6f6b142d0ee5d77425468261544611c1a0" exitCode=0 Nov 03 01:19:17.642886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:17.642479 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" event={"ID":"3941e883-dd6c-40e4-911a-8c6f46455248","Type":"ContainerDied","Data":"dceca7f152f8c15bd16dab37ff5b8b6f6b142d0ee5d77425468261544611c1a0"} Nov 03 01:19:18.815809 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.815781 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:18.939130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.939005 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca\") pod \"3941e883-dd6c-40e4-911a-8c6f46455248\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " Nov 03 01:19:18.939308 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.939230 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df\" (UniqueName: \"kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df\") pod \"3941e883-dd6c-40e4-911a-8c6f46455248\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " Nov 03 01:19:18.939394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.939318 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca\") pod \"3941e883-dd6c-40e4-911a-8c6f46455248\" (UID: \"3941e883-dd6c-40e4-911a-8c6f46455248\") " Nov 03 01:19:18.940459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.940435 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca" pod "3941e883-dd6c-40e4-911a-8c6f46455248" (UID: "3941e883-dd6c-40e4-911a-8c6f46455248"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:18.942313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.942286 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca" pod "3941e883-dd6c-40e4-911a-8c6f46455248" (UID: "3941e883-dd6c-40e4-911a-8c6f46455248"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:18.942657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:18.942616 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df" pod "3941e883-dd6c-40e4-911a-8c6f46455248" (UID: "3941e883-dd6c-40e4-911a-8c6f46455248"). InnerVolumeSpecName "kube-api-access-2c5df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:19.041619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.041578 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c5df\" (UniqueName: \"kubernetes.io/projected/3941e883-dd6c-40e4-911a-8c6f46455248-kube-api-access-2c5df\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:19.041619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.041614 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:19.041619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.041643 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3941e883-dd6c-40e4-911a-8c6f46455248-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:19.660164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.660137 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" Nov 03 01:19:19.660164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.660149 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9" event={"ID":"3941e883-dd6c-40e4-911a-8c6f46455248","Type":"ContainerDied","Data":"7d2fabb35ee9eff825dc49cefdf5a90f7464514478d5003c7150f7f1c7d985e4"} Nov 03 01:19:19.660429 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.660213 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d2fabb35ee9eff825dc49cefdf5a90f7464514478d5003c7150f7f1c7d985e4" Nov 03 01:19:19.964901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:19.964870 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 01:19:20.052924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.052889 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume\") pod \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " Nov 03 01:19:20.053123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.052973 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh\" (UniqueName: \"kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh\") pod \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\" (UID: \"05c39946-84c5-42f5-9dc4-ca66cdebea2b\") " Nov 03 01:19:20.053675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.053642 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume" pod "05c39946-84c5-42f5-9dc4-ca66cdebea2b" (UID: "05c39946-84c5-42f5-9dc4-ca66cdebea2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:20.056510 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.056475 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh" pod "05c39946-84c5-42f5-9dc4-ca66cdebea2b" (UID: "05c39946-84c5-42f5-9dc4-ca66cdebea2b"). InnerVolumeSpecName "kube-api-access-s4kvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:20.091892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.091826 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 01:19:20.153719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.153687 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume\") pod \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " Nov 03 01:19:20.153908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.153759 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm\" (UniqueName: \"kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm\") pod \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\" (UID: \"5b5e6d74-afac-4955-9634-c6f6374d0bd3\") " Nov 03 01:19:20.154016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.154001 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/05c39946-84c5-42f5-9dc4-ca66cdebea2b-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.154089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.154017 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4kvh\" (UniqueName: \"kubernetes.io/projected/05c39946-84c5-42f5-9dc4-ca66cdebea2b-kube-api-access-s4kvh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.154089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.154049 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume" pod "5b5e6d74-afac-4955-9634-c6f6374d0bd3" (UID: "5b5e6d74-afac-4955-9634-c6f6374d0bd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:20.156088 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.156063 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm" pod "5b5e6d74-afac-4955-9634-c6f6374d0bd3" (UID: "5b5e6d74-afac-4955-9634-c6f6374d0bd3"). InnerVolumeSpecName "kube-api-access-b47bm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:20.255013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.254965 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b5e6d74-afac-4955-9634-c6f6374d0bd3-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.255178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.255017 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b47bm\" (UniqueName: \"kubernetes.io/projected/5b5e6d74-afac-4955-9634-c6f6374d0bd3-kube-api-access-b47bm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:20.670602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.670509 5144 generic.go:356] "Generic (PLEG): container finished" podID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerID="54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e" exitCode=0 Nov 03 01:19:20.670602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.670586 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" Nov 03 01:19:20.670887 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.670603 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" event={"ID":"05c39946-84c5-42f5-9dc4-ca66cdebea2b","Type":"ContainerDied","Data":"54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e"} Nov 03 01:19:20.670887 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.670735 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg" event={"ID":"05c39946-84c5-42f5-9dc4-ca66cdebea2b","Type":"ContainerDied","Data":"b60ef14f1751e35015cf0ac89884ce96dc82a7074591589bcf3ef1fa0ccefaf4"} Nov 03 01:19:20.670887 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.670759 5144 scope.go:122] "RemoveContainer" containerID="54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e" Nov 03 01:19:20.672355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.672331 5144 generic.go:356] "Generic (PLEG): container finished" podID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerID="03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3" exitCode=0 Nov 03 01:19:20.672482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.672395 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" event={"ID":"5b5e6d74-afac-4955-9634-c6f6374d0bd3","Type":"ContainerDied","Data":"03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3"} Nov 03 01:19:20.672482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.672401 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" Nov 03 01:19:20.672482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.672433 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7" event={"ID":"5b5e6d74-afac-4955-9634-c6f6374d0bd3","Type":"ContainerDied","Data":"2820b2fc896ad8c7a6fc8f97dbbde717366c687beda7103eca7a3cf3957590a7"} Nov 03 01:19:20.685088 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.685073 5144 scope.go:122] "RemoveContainer" containerID="54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e" Nov 03 01:19:20.685383 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:20.685352 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e\": container with ID starting with 54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e not found: ID does not exist" containerID="54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e" Nov 03 01:19:20.685492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.685393 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e"} err="failed to get container status \"54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e\": rpc error: code = NotFound desc = could not find container \"54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e\": container with ID starting with 54bf53eb28c8c2f574642c110f41efa6ada8bfa4ccf9947c610425e34cea7b9e not found: ID does not exist" Nov 03 01:19:20.685492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.685415 5144 scope.go:122] "RemoveContainer" containerID="03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3" Nov 03 01:19:20.697034 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.697018 5144 scope.go:122] "RemoveContainer" containerID="03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3" Nov 03 01:19:20.697319 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:20.697300 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3\": container with ID starting with 03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3 not found: ID does not exist" containerID="03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3" Nov 03 01:19:20.697383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.697325 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3"} err="failed to get container status \"03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3\": rpc error: code = NotFound desc = could not find container \"03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3\": container with ID starting with 03f603c8bad58b7d534ab57cc4e1baecef9779c36f96a32a2725f05e7e29b9f3 not found: ID does not exist" Nov 03 01:19:20.835100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.835075 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg"] Nov 03 01:19:20.884623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.884595 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-328e1b-68c9f547c7-kvxbg"] Nov 03 01:19:20.954316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.954226 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7"] Nov 03 01:19:20.957590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:20.957564 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-328e1b-6648b94747-j8jr7"] Nov 03 01:19:21.997017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:21.996984 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" path="/var/lib/kubelet/pods/05c39946-84c5-42f5-9dc4-ca66cdebea2b/volumes" Nov 03 01:19:21.997466 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:21.997446 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" path="/var/lib/kubelet/pods/5b5e6d74-afac-4955-9634-c6f6374d0bd3/volumes" Nov 03 01:19:23.412901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.412871 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs"] Nov 03 01:19:23.413423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413410 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerName="payload-pre-processing" Nov 03 01:19:23.413469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413425 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerName="payload-pre-processing" Nov 03 01:19:23.413469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413453 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3941e883-dd6c-40e4-911a-8c6f46455248" containerName="revoke-keys" Nov 03 01:19:23.413469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413458 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3941e883-dd6c-40e4-911a-8c6f46455248" containerName="revoke-keys" Nov 03 01:19:23.413563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413472 5144 cpu_manager.go:414] "containerMap: removing container" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerName="payload-processing" Nov 03 01:19:23.413563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413477 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerName="payload-processing" Nov 03 01:19:23.413563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413555 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3941e883-dd6c-40e4-911a-8c6f46455248" containerName="revoke-keys" Nov 03 01:19:23.413705 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413563 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c39946-84c5-42f5-9dc4-ca66cdebea2b" containerName="payload-processing" Nov 03 01:19:23.413705 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.413575 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5e6d74-afac-4955-9634-c6f6374d0bd3" containerName="payload-pre-processing" Nov 03 01:19:23.418458 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.418429 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.473886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.473862 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-d7bfc2-openshift-default-dockercfg-g56sm\"" Nov 03 01:19:23.487495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487468 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487516 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487605 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487751 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487697 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487751 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487728 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487769 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487822 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487911 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487855 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.487944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.487923 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5m9\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.523865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.523836 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs"] Nov 03 01:19:23.588774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588741 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.588962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588793 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.588962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588815 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.588962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588843 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.588962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588876 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.588962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588910 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.588962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588950 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.589353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588971 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.589353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.588998 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5m9\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.589353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.589197 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.589525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.589459 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.589583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.589558 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.589677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.589588 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.590153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.590125 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.591298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.591279 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.591745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.591728 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.645586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.645563 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.645741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.645567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5m9\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9\") pod \"e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:23.733830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:23.733760 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:24.160434 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:24.160406 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddda2f499_fe72_44a9_8cb4_23b7578e835e.slice/crio-869de8a7f8d2f4c8b9f726a46e7a2aec1a99be7085d1ee9be6979f77cf7e4af6 WatchSource:0}: Error finding container 869de8a7f8d2f4c8b9f726a46e7a2aec1a99be7085d1ee9be6979f77cf7e4af6: Status 404 returned error can't find the container with id 869de8a7f8d2f4c8b9f726a46e7a2aec1a99be7085d1ee9be6979f77cf7e4af6 Nov 03 01:19:24.162781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.162758 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs"] Nov 03 01:19:24.163114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.163084 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:19:24.163174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.163150 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:19:24.163211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.163179 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:19:24.704767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.704729 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" event={"ID":"dda2f499-fe72-44a9-8cb4-23b7578e835e","Type":"ContainerStarted","Data":"bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a"} Nov 03 01:19:24.705239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.704788 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" event={"ID":"dda2f499-fe72-44a9-8cb4-23b7578e835e","Type":"ContainerStarted","Data":"869de8a7f8d2f4c8b9f726a46e7a2aec1a99be7085d1ee9be6979f77cf7e4af6"} Nov 03 01:19:24.738659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:24.733926 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:25.742656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:25.740512 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:19:25.742656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:25.740571 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:19:26.737901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:26.737853 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:19:26.738209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:26.738176 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:19:27.735224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:27.735173 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:19:27.735859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:27.735811 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:19:28.740996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:28.740961 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:19:28.741608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:28.741572 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:19:29.734373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:29.734331 5144 patch_prober.go:29] interesting pod/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" start-of-body= Nov 03 01:19:29.734709 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:29.734682 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.86:15021/healthz/ready\": dial tcp 10.132.1.86:15021: connect: connection refused" Nov 03 01:19:30.326623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:30.326560 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podStartSLOduration=7.326538514 podStartE2EDuration="7.326538514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:24.829005892 +0000 UTC m=+4454.974700364" watchObservedRunningTime="2026-11-03 01:19:30.326538514 +0000 UTC m=+4460.472232971" Nov 03 01:19:30.329385 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:30.329362 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs"] Nov 03 01:19:30.329603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:30.329581 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" containerID="cri-o://bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a" gracePeriod=30 Nov 03 01:19:36.383462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.383435 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:36.451601 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451532 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451601 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451577 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451643 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451727 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451768 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451827 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451909 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451941 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.451992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451967 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9\") pod \"dda2f499-fe72-44a9-8cb4-23b7578e835e\" (UID: \"dda2f499-fe72-44a9-8cb4-23b7578e835e\") " Nov 03 01:19:36.452147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451820 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:36.452147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451945 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:36.452147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.451970 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:36.452147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452087 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:36.452296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452186 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:36.452498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452477 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda2f499-fe72-44a9-8cb4-23b7578e835e-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.452667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452501 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.452667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452514 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.452667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452527 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.452667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.452539 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.454452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.454426 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "kube-api-access-lj5m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.454570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.454532 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:19:36.454670 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.454588 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:36.454739 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.454693 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token" pod "dda2f499-fe72-44a9-8cb4-23b7578e835e" (UID: "dda2f499-fe72-44a9-8cb4-23b7578e835e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:36.553795 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.553766 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.553795 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.553797 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.554002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.553811 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5m9\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-kube-api-access-lj5m9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.554002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.553826 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda2f499-fe72-44a9-8cb4-23b7578e835e-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:36.817369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.817332 5144 generic.go:356] "Generic (PLEG): container finished" podID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerID="bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a" exitCode=0 Nov 03 01:19:36.817563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.817384 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" Nov 03 01:19:36.817563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.817418 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" event={"ID":"dda2f499-fe72-44a9-8cb4-23b7578e835e","Type":"ContainerDied","Data":"bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a"} Nov 03 01:19:36.817563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.817484 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs" event={"ID":"dda2f499-fe72-44a9-8cb4-23b7578e835e","Type":"ContainerDied","Data":"869de8a7f8d2f4c8b9f726a46e7a2aec1a99be7085d1ee9be6979f77cf7e4af6"} Nov 03 01:19:36.817563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.817515 5144 scope.go:122] "RemoveContainer" containerID="bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a" Nov 03 01:19:36.843383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.843335 5144 scope.go:122] "RemoveContainer" containerID="bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a" Nov 03 01:19:36.844208 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:36.844177 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a\": container with ID starting with bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a not found: ID does not exist" containerID="bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a" Nov 03 01:19:36.844337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.844218 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a"} err="failed to get container status \"bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a\": rpc error: code = NotFound desc = could not find container \"bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a\": container with ID starting with bddcaacdd4ef2320701a151cebdccb6e13cf4f41e9322ef012c9b2bec30aee3a not found: ID does not exist" Nov 03 01:19:36.994433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.994382 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs"] Nov 03 01:19:36.997534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:36.997502 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-d7bfc2-openshift-default-85dbf88679-ggrrs"] Nov 03 01:19:37.997047 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:37.997017 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" path="/var/lib/kubelet/pods/dda2f499-fe72-44a9-8cb4-23b7578e835e/volumes" Nov 03 01:19:42.162678 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:42.162619 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:19:43.973855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:43.973821 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7297-8c49e4"] Nov 03 01:19:43.974408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:43.974393 5144 cpu_manager.go:414] "containerMap: removing container" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" Nov 03 01:19:43.974451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:43.974408 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" Nov 03 01:19:43.974498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:43.974489 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda2f499-fe72-44a9-8cb4-23b7578e835e" containerName="istio-proxy" Nov 03 01:19:43.979332 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:43.979311 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:44.031886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.031863 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-8c49e4"] Nov 03 01:19:44.039484 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.039454 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd8r2\" (UniqueName: \"kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2\") pod \"test-curl-7297-8c49e4\" (UID: \"cf32209b-2a22-43bd-b701-3a3bc94ee012\") " pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:44.077359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.077336 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-fz96m\"" Nov 03 01:19:44.140296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.140268 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd8r2\" (UniqueName: \"kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2\") pod \"test-curl-7297-8c49e4\" (UID: \"cf32209b-2a22-43bd-b701-3a3bc94ee012\") " pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:44.196904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.196873 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd8r2\" (UniqueName: \"kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2\") pod \"test-curl-7297-8c49e4\" (UID: \"cf32209b-2a22-43bd-b701-3a3bc94ee012\") " pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:44.291392 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.291364 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:44.670609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.670581 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-8c49e4"] Nov 03 01:19:44.671445 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:44.671419 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf32209b_2a22_43bd_b701_3a3bc94ee012.slice/crio-ec791033c55c798de36eba310f12eb5ea5e1d909320d6cd23b171c728084c2f3 WatchSource:0}: Error finding container ec791033c55c798de36eba310f12eb5ea5e1d909320d6cd23b171c728084c2f3: Status 404 returned error can't find the container with id ec791033c55c798de36eba310f12eb5ea5e1d909320d6cd23b171c728084c2f3 Nov 03 01:19:44.875190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.875143 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-8c49e4" event={"ID":"cf32209b-2a22-43bd-b701-3a3bc94ee012","Type":"ContainerStarted","Data":"bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174"} Nov 03 01:19:44.875436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.875200 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-8c49e4" event={"ID":"cf32209b-2a22-43bd-b701-3a3bc94ee012","Type":"ContainerStarted","Data":"ec791033c55c798de36eba310f12eb5ea5e1d909320d6cd23b171c728084c2f3"} Nov 03 01:19:44.987148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:44.987054 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7297-8c49e4" podStartSLOduration=1.9870387680000001 podStartE2EDuration="1.987038768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:44.984877201 +0000 UTC m=+4475.130571673" watchObservedRunningTime="2026-11-03 01:19:44.987038768 +0000 UTC m=+4475.132733224" Nov 03 01:19:46.373356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.373324 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd"] Nov 03 01:19:46.373858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.373508 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" containerID="cri-o://042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f" gracePeriod=30 Nov 03 01:19:46.426556 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.426520 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7297-8c49e4"] Nov 03 01:19:46.431265 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.431240 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7297-8c49e4"] Nov 03 01:19:46.572956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.572924 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7297-87748f"] Nov 03 01:19:46.573472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.573459 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" containerName="test-curl-7297-8c49e4" Nov 03 01:19:46.573533 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.573474 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" containerName="test-curl-7297-8c49e4" Nov 03 01:19:46.573585 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.573576 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" containerName="test-curl-7297-8c49e4" Nov 03 01:19:46.577039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.577021 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:46.628334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.628280 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-87748f"] Nov 03 01:19:46.668086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.668054 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcv4w\" (UniqueName: \"kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w\") pod \"test-curl-7297-87748f\" (UID: \"7e9c1431-a720-4f39-9bbd-870759816d90\") " pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:46.769770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.769738 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcv4w\" (UniqueName: \"kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w\") pod \"test-curl-7297-87748f\" (UID: \"7e9c1431-a720-4f39-9bbd-870759816d90\") " pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:46.825523 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.825491 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcv4w\" (UniqueName: \"kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w\") pod \"test-curl-7297-87748f\" (UID: \"7e9c1431-a720-4f39-9bbd-870759816d90\") " pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:46.888131 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.888066 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7297-8c49e4" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" containerName="test-curl-7297-8c49e4" containerID="cri-o://bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174" gracePeriod=2 Nov 03 01:19:46.889452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:46.889426 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:46.938673 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:46.938609 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-8c49e4\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:47.262474 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.262442 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7297-87748f"] Nov 03 01:19:47.270084 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:47.270053 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e9c1431_a720_4f39_9bbd_870759816d90.slice/crio-34108ef99d7bd68152382a7499e57b9dfded99e296884c84ccf006d908537230 WatchSource:0}: Error finding container 34108ef99d7bd68152382a7499e57b9dfded99e296884c84ccf006d908537230: Status 404 returned error can't find the container with id 34108ef99d7bd68152382a7499e57b9dfded99e296884c84ccf006d908537230 Nov 03 01:19:47.477755 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.477731 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4"] Nov 03 01:19:47.481685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.481670 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.532836 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:47.532786 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-8c49e4\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:47.534348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.534326 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:19:47.534516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.534450 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:19:47.537231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.537211 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4"] Nov 03 01:19:47.578686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.578662 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.578799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.578708 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.578799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.578726 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhnh\" (UniqueName: \"kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.679712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.679687 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.679818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.679730 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.679818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.679747 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhnh\" (UniqueName: \"kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.680379 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.680361 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.682340 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.682321 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.735662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.735623 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhnh\" (UniqueName: \"kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh\") pod \"maas-api-revoke-sub-af0e65ce-qh2b4\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.794995 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.794943 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:47.902491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.902452 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-87748f" event={"ID":"7e9c1431-a720-4f39-9bbd-870759816d90","Type":"ContainerStarted","Data":"156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc"} Nov 03 01:19:47.902717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:47.902526 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7297-87748f" event={"ID":"7e9c1431-a720-4f39-9bbd-870759816d90","Type":"ContainerStarted","Data":"34108ef99d7bd68152382a7499e57b9dfded99e296884c84ccf006d908537230"} Nov 03 01:19:47.953180 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:47.953145 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-8c49e4\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:48.066725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:48.066619 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7297-87748f" podStartSLOduration=2.066606031 podStartE2EDuration="2.066606031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:48.063885702 +0000 UTC m=+4478.209580171" watchObservedRunningTime="2026-11-03 01:19:48.066606031 +0000 UTC m=+4478.212300486" Nov 03 01:19:48.166802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:48.166775 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4"] Nov 03 01:19:48.166997 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:48.166977 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38720dd6_b1e2_4108_9b3e_b59ff45f68d6.slice/crio-592573fbbba3097f13dc61832a9bb4fa69784d143d2c7392837fe6fdc64047c1 WatchSource:0}: Error finding container 592573fbbba3097f13dc61832a9bb4fa69784d143d2c7392837fe6fdc64047c1: Status 404 returned error can't find the container with id 592573fbbba3097f13dc61832a9bb4fa69784d143d2c7392837fe6fdc64047c1 Nov 03 01:19:48.910911 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:48.910873 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" event={"ID":"38720dd6-b1e2-4108-9b3e-b59ff45f68d6","Type":"ContainerStarted","Data":"9d89b1db16e525ec53b5e0bfe614270e8c0e46128949646dd06d732c75def8cf"} Nov 03 01:19:48.911682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:48.910943 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" event={"ID":"38720dd6-b1e2-4108-9b3e-b59ff45f68d6","Type":"ContainerStarted","Data":"592573fbbba3097f13dc61832a9bb4fa69784d143d2c7392837fe6fdc64047c1"} Nov 03 01:19:49.076042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.075996 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" podStartSLOduration=2.075982416 podStartE2EDuration="2.075982416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:49.069950845 +0000 UTC m=+4479.215645312" watchObservedRunningTime="2026-11-03 01:19:49.075982416 +0000 UTC m=+4479.221676872" Nov 03 01:19:49.141140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.141117 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:49.298234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.298206 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2\" (UniqueName: \"kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2\") pod \"cf32209b-2a22-43bd-b701-3a3bc94ee012\" (UID: \"cf32209b-2a22-43bd-b701-3a3bc94ee012\") " Nov 03 01:19:49.300276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.300251 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2" pod "cf32209b-2a22-43bd-b701-3a3bc94ee012" (UID: "cf32209b-2a22-43bd-b701-3a3bc94ee012"). InnerVolumeSpecName "kube-api-access-rd8r2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:49.399104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.399077 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd8r2\" (UniqueName: \"kubernetes.io/projected/cf32209b-2a22-43bd-b701-3a3bc94ee012-kube-api-access-rd8r2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:49.507409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.507384 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7297-87748f"] Nov 03 01:19:49.511058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.511033 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7297-87748f"] Nov 03 01:19:49.918249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.918219 5144 generic.go:356] "Generic (PLEG): container finished" podID="cf32209b-2a22-43bd-b701-3a3bc94ee012" containerID="bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174" exitCode=137 Nov 03 01:19:49.918757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.918261 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-8c49e4" Nov 03 01:19:49.918757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.918319 5144 scope.go:122] "RemoveContainer" containerID="bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174" Nov 03 01:19:49.920362 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.920326 5144 generic.go:356] "Generic (PLEG): container finished" podID="38720dd6-b1e2-4108-9b3e-b59ff45f68d6" containerID="9d89b1db16e525ec53b5e0bfe614270e8c0e46128949646dd06d732c75def8cf" exitCode=0 Nov 03 01:19:49.920476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.920444 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" event={"ID":"38720dd6-b1e2-4108-9b3e-b59ff45f68d6","Type":"ContainerDied","Data":"9d89b1db16e525ec53b5e0bfe614270e8c0e46128949646dd06d732c75def8cf"} Nov 03 01:19:49.920624 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.920604 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7297-87748f" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" containerName="test-curl-7297-87748f" containerID="cri-o://156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc" gracePeriod=2 Nov 03 01:19:49.930210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.930193 5144 scope.go:122] "RemoveContainer" containerID="bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174" Nov 03 01:19:49.930518 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:49.930494 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174\": container with ID starting with bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174 not found: ID does not exist" containerID="bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174" Nov 03 01:19:49.930588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.930528 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174"} err="failed to get container status \"bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174\": rpc error: code = NotFound desc = could not find container \"bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174\": container with ID starting with bb44ca9cda1962ca6fc2181604156fde8ace019836aeaf181e39c5a63d06e174 not found: ID does not exist" Nov 03 01:19:49.996286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:49.996263 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf32209b-2a22-43bd-b701-3a3bc94ee012" path="/var/lib/kubelet/pods/cf32209b-2a22-43bd-b701-3a3bc94ee012/volumes" Nov 03 01:19:50.546157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.546121 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l"] Nov 03 01:19:50.546754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.546735 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" containerName="test-curl-7297-87748f" Nov 03 01:19:50.546754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.546753 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" containerName="test-curl-7297-87748f" Nov 03 01:19:50.546944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.546876 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" containerName="test-curl-7297-87748f" Nov 03 01:19:50.551682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.551660 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.594820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.594790 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l"] Nov 03 01:19:50.714419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.714387 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbrx\" (UniqueName: \"kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.714600 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.714447 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.714688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.714624 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.816121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.816051 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.816121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.816095 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbrx\" (UniqueName: \"kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.816291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.816126 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.816807 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.816783 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.818501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.818474 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:50.872293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:50.872261 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbrx\" (UniqueName: \"kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx\") pod \"maas-api-revoke-sub-725b559c-lt98l\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:51.078806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.078745 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:51.169917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.169890 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:51.220341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.220305 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca\") pod \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " Nov 03 01:19:51.220522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.220386 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca\") pod \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " Nov 03 01:19:51.220522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.220470 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh\" (UniqueName: \"kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh\") pod \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\" (UID: \"38720dd6-b1e2-4108-9b3e-b59ff45f68d6\") " Nov 03 01:19:51.221177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.221089 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca" pod "38720dd6-b1e2-4108-9b3e-b59ff45f68d6" (UID: "38720dd6-b1e2-4108-9b3e-b59ff45f68d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:51.222781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.222755 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca" pod "38720dd6-b1e2-4108-9b3e-b59ff45f68d6" (UID: "38720dd6-b1e2-4108-9b3e-b59ff45f68d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:51.223274 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.223252 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh" pod "38720dd6-b1e2-4108-9b3e-b59ff45f68d6" (UID: "38720dd6-b1e2-4108-9b3e-b59ff45f68d6"). InnerVolumeSpecName "kube-api-access-zfhnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:51.322128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.322087 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.322128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.322125 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.322128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.322136 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhnh\" (UniqueName: \"kubernetes.io/projected/38720dd6-b1e2-4108-9b3e-b59ff45f68d6-kube-api-access-zfhnh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.554537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.552203 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l"] Nov 03 01:19:51.554537 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:19:51.553931 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod948b3157_b4f3_4486_a016_4322047d4b88.slice/crio-8718af3cb482c428e4228a44c24f7c825b66f86408ebeb0f9e7fbf548c1b3b88 WatchSource:0}: Error finding container 8718af3cb482c428e4228a44c24f7c825b66f86408ebeb0f9e7fbf548c1b3b88: Status 404 returned error can't find the container with id 8718af3cb482c428e4228a44c24f7c825b66f86408ebeb0f9e7fbf548c1b3b88 Nov 03 01:19:51.612735 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.612712 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 01:19:51.726535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726510 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726704 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726607 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726704 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726660 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726704 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726695 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726730 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726768 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726819 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726874 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:51.726967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726898 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.726967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726953 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy\") pod \"f46ed093-2590-4248-ad19-409dd68099a1\" (UID: \"f46ed093-2590-4248-ad19-409dd68099a1\") " Nov 03 01:19:51.727289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.726970 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:51.727289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.727096 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:51.727289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.727204 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:51.727443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.727402 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.728860 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.728831 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:51.730073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.730052 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.730175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.730083 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.730175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.730098 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.731588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.731333 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "kube-api-access-hhvp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:51.731843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.731823 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:19:51.733824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.733419 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:19:51.734452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.734003 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token" pod "f46ed093-2590-4248-ad19-409dd68099a1" (UID: "f46ed093-2590-4248-ad19-409dd68099a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:51.797592 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.797566 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq"] Nov 03 01:19:51.814496 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.814470 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn"] Nov 03 01:19:51.831838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.831815 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f46ed093-2590-4248-ad19-409dd68099a1-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.831838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.831838 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhvp8\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-kube-api-access-hhvp8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.832015 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.831849 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f46ed093-2590-4248-ad19-409dd68099a1-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.832015 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.831857 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f46ed093-2590-4248-ad19-409dd68099a1-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.832015 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.831865 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f46ed093-2590-4248-ad19-409dd68099a1-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:51.939051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.939026 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" Nov 03 01:19:51.939203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.939032 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4" event={"ID":"38720dd6-b1e2-4108-9b3e-b59ff45f68d6","Type":"ContainerDied","Data":"592573fbbba3097f13dc61832a9bb4fa69784d143d2c7392837fe6fdc64047c1"} Nov 03 01:19:51.939203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.939150 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="592573fbbba3097f13dc61832a9bb4fa69784d143d2c7392837fe6fdc64047c1" Nov 03 01:19:51.941064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.941042 5144 generic.go:356] "Generic (PLEG): container finished" podID="f46ed093-2590-4248-ad19-409dd68099a1" containerID="042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f" exitCode=0 Nov 03 01:19:51.941334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.941085 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" Nov 03 01:19:51.941334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.941119 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" event={"ID":"f46ed093-2590-4248-ad19-409dd68099a1","Type":"ContainerDied","Data":"042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f"} Nov 03 01:19:51.941334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.941176 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd" event={"ID":"f46ed093-2590-4248-ad19-409dd68099a1","Type":"ContainerDied","Data":"856c2694da6cd8e24825baf918959c49ddf4234a462503f83a692dc8fbcc7090"} Nov 03 01:19:51.941334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.941197 5144 scope.go:122] "RemoveContainer" containerID="042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f" Nov 03 01:19:51.942996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.942974 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" event={"ID":"948b3157-b4f3-4486-a016-4322047d4b88","Type":"ContainerStarted","Data":"3018253a96c52b601abaeac2bbecda9ea759284b8b81a8c7f27f7c346f3fb52f"} Nov 03 01:19:51.943093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.943017 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" event={"ID":"948b3157-b4f3-4486-a016-4322047d4b88","Type":"ContainerStarted","Data":"8718af3cb482c428e4228a44c24f7c825b66f86408ebeb0f9e7fbf548c1b3b88"} Nov 03 01:19:51.982520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.982494 5144 scope.go:122] "RemoveContainer" containerID="75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93" Nov 03 01:19:51.994767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.994745 5144 scope.go:122] "RemoveContainer" containerID="042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f" Nov 03 01:19:51.995050 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:51.995025 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f\": container with ID starting with 042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f not found: ID does not exist" containerID="042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f" Nov 03 01:19:51.995137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.995056 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f"} err="failed to get container status \"042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f\": rpc error: code = NotFound desc = could not find container \"042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f\": container with ID starting with 042a32e514e8d35f0f3fd3d503021413df9541471a6f942003872717c063569f not found: ID does not exist" Nov 03 01:19:51.995137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.995075 5144 scope.go:122] "RemoveContainer" containerID="75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93" Nov 03 01:19:51.995386 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:51.995361 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93\": container with ID starting with 75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93 not found: ID does not exist" containerID="75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93" Nov 03 01:19:51.995492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:51.995393 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93"} err="failed to get container status \"75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93\": rpc error: code = NotFound desc = could not find container \"75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93\": container with ID starting with 75b2532b73f0701c1394e830477cd8bb72c69cfedcee6feed8ef7fb377f07b93 not found: ID does not exist" Nov 03 01:19:52.141858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.137748 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd"] Nov 03 01:19:52.142534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.142511 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-328e1b-openshift-default-74cfd7d98c-7rgjd"] Nov 03 01:19:52.148623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.148601 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:52.251373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.251277 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" podStartSLOduration=2.251265074 podStartE2EDuration="2.251265074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:52.250944439 +0000 UTC m=+4482.396638905" watchObservedRunningTime="2026-11-03 01:19:52.251265074 +0000 UTC m=+4482.396959528" Nov 03 01:19:52.337502 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.337478 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w\" (UniqueName: \"kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w\") pod \"7e9c1431-a720-4f39-9bbd-870759816d90\" (UID: \"7e9c1431-a720-4f39-9bbd-870759816d90\") " Nov 03 01:19:52.339668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.339645 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w" pod "7e9c1431-a720-4f39-9bbd-870759816d90" (UID: "7e9c1431-a720-4f39-9bbd-870759816d90"). InnerVolumeSpecName "kube-api-access-bcv4w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:52.352794 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:52.352767 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-87748f\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:52.438596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.438565 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcv4w\" (UniqueName: \"kubernetes.io/projected/7e9c1431-a720-4f39-9bbd-870759816d90-kube-api-access-bcv4w\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:52.952269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.952233 5144 generic.go:356] "Generic (PLEG): container finished" podID="7e9c1431-a720-4f39-9bbd-870759816d90" containerID="156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc" exitCode=137 Nov 03 01:19:52.952459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.952281 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:52.952459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.952329 5144 scope.go:122] "RemoveContainer" containerID="156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc" Nov 03 01:19:52.954237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.954213 5144 generic.go:356] "Generic (PLEG): container finished" podID="948b3157-b4f3-4486-a016-4322047d4b88" containerID="3018253a96c52b601abaeac2bbecda9ea759284b8b81a8c7f27f7c346f3fb52f" exitCode=0 Nov 03 01:19:52.954341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.954270 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" event={"ID":"948b3157-b4f3-4486-a016-4322047d4b88","Type":"ContainerDied","Data":"3018253a96c52b601abaeac2bbecda9ea759284b8b81a8c7f27f7c346f3fb52f"} Nov 03 01:19:52.963716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.963699 5144 scope.go:122] "RemoveContainer" containerID="156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc" Nov 03 01:19:52.963997 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:52.963974 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc\": container with ID starting with 156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc not found: ID does not exist" containerID="156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc" Nov 03 01:19:52.964049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:52.964009 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc"} err="failed to get container status \"156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc\": rpc error: code = NotFound desc = could not find container \"156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc\": container with ID starting with 156977463c143557d3c969f22f5ac5bdbab7a4b5c706c832f086770b0d49c1dc not found: ID does not exist" Nov 03 01:19:53.002950 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:53.002914 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-87748f\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:53.053519 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:19:53.053492 5144 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7297-87748f\" is forbidden: User \"system:node:ip-10-0-143-162.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-143-162.us-west-1.compute.internal' and this object" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" pod="opendatahub/test-curl-7297-87748f" Nov 03 01:19:54.003207 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.003173 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e9c1431-a720-4f39-9bbd-870759816d90" path="/var/lib/kubelet/pods/7e9c1431-a720-4f39-9bbd-870759816d90/volumes" Nov 03 01:19:54.003984 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.003650 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46ed093-2590-4248-ad19-409dd68099a1" path="/var/lib/kubelet/pods/f46ed093-2590-4248-ad19-409dd68099a1/volumes" Nov 03 01:19:54.107480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.107457 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:54.155588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.155565 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca\") pod \"948b3157-b4f3-4486-a016-4322047d4b88\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " Nov 03 01:19:54.155761 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.155620 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx\" (UniqueName: \"kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx\") pod \"948b3157-b4f3-4486-a016-4322047d4b88\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " Nov 03 01:19:54.155761 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.155674 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca\") pod \"948b3157-b4f3-4486-a016-4322047d4b88\" (UID: \"948b3157-b4f3-4486-a016-4322047d4b88\") " Nov 03 01:19:54.156079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.156048 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca" pod "948b3157-b4f3-4486-a016-4322047d4b88" (UID: "948b3157-b4f3-4486-a016-4322047d4b88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:19:54.157879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.157858 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca" pod "948b3157-b4f3-4486-a016-4322047d4b88" (UID: "948b3157-b4f3-4486-a016-4322047d4b88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:19:54.157879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.157873 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx" pod "948b3157-b4f3-4486-a016-4322047d4b88" (UID: "948b3157-b4f3-4486-a016-4322047d4b88"). InnerVolumeSpecName "kube-api-access-mcbrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:19:54.257262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.257241 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbrx\" (UniqueName: \"kubernetes.io/projected/948b3157-b4f3-4486-a016-4322047d4b88-kube-api-access-mcbrx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.257374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.257264 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.257374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.257274 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/948b3157-b4f3-4486-a016-4322047d4b88-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:19:54.971422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.971391 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" Nov 03 01:19:54.971605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.971421 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l" event={"ID":"948b3157-b4f3-4486-a016-4322047d4b88","Type":"ContainerDied","Data":"8718af3cb482c428e4228a44c24f7c825b66f86408ebeb0f9e7fbf548c1b3b88"} Nov 03 01:19:54.971605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:19:54.971498 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8718af3cb482c428e4228a44c24f7c825b66f86408ebeb0f9e7fbf548c1b3b88" Nov 03 01:20:01.296917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.296816 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj"] Nov 03 01:20:01.297365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297351 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" Nov 03 01:20:01.297410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297366 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" Nov 03 01:20:01.297410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297379 5144 cpu_manager.go:414] "containerMap: removing container" podUID="948b3157-b4f3-4486-a016-4322047d4b88" containerName="revoke-keys" Nov 03 01:20:01.297410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297388 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="948b3157-b4f3-4486-a016-4322047d4b88" containerName="revoke-keys" Nov 03 01:20:01.297516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297434 5144 cpu_manager.go:414] "containerMap: removing container" podUID="38720dd6-b1e2-4108-9b3e-b59ff45f68d6" containerName="revoke-keys" Nov 03 01:20:01.297516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297439 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38720dd6-b1e2-4108-9b3e-b59ff45f68d6" containerName="revoke-keys" Nov 03 01:20:01.297516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297513 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="38720dd6-b1e2-4108-9b3e-b59ff45f68d6" containerName="revoke-keys" Nov 03 01:20:01.297608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297523 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" Nov 03 01:20:01.297608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.297528 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="948b3157-b4f3-4486-a016-4322047d4b88" containerName="revoke-keys" Nov 03 01:20:01.300832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.300816 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.329699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.325428 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.329699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.325543 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.329699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.325700 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-879l7\" (UniqueName: \"kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.355370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.355322 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:20:01.404867 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.404842 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:20:01.408177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.408157 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj"] Nov 03 01:20:01.426811 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.426779 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-879l7\" (UniqueName: \"kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.426811 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.426813 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.427016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.426861 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.427584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.427565 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.429469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.429445 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.482335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.482313 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-879l7\" (UniqueName: \"kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7\") pod \"maas-api-revoke-sub-06ee0c2f-h9wxj\" (UID: \"16362872-a310-4406-8534-117656535136\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.614297 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.614220 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:01.992888 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:20:01.992859 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16362872_a310_4406_8534_117656535136.slice/crio-410bd7c14aa9ae1c079c7cb9265d71182de47c846fcb742301d6262337f780f3 WatchSource:0}: Error finding container 410bd7c14aa9ae1c079c7cb9265d71182de47c846fcb742301d6262337f780f3: Status 404 returned error can't find the container with id 410bd7c14aa9ae1c079c7cb9265d71182de47c846fcb742301d6262337f780f3 Nov 03 01:20:01.998194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:01.998164 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj"] Nov 03 01:20:02.035199 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:02.035155 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" event={"ID":"16362872-a310-4406-8534-117656535136","Type":"ContainerStarted","Data":"410bd7c14aa9ae1c079c7cb9265d71182de47c846fcb742301d6262337f780f3"} Nov 03 01:20:03.042979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.042947 5144 generic.go:356] "Generic (PLEG): container finished" podID="16362872-a310-4406-8534-117656535136" containerID="c111d6c90605eb24edea7608e6774f390e0ab818e1146815ad51bb7a381f4e4a" exitCode=0 Nov 03 01:20:03.043388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.043036 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" event={"ID":"16362872-a310-4406-8534-117656535136","Type":"ContainerDied","Data":"c111d6c90605eb24edea7608e6774f390e0ab818e1146815ad51bb7a381f4e4a"} Nov 03 01:20:03.515233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.515201 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn"] Nov 03 01:20:03.621711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.621683 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn"] Nov 03 01:20:03.996179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.996106 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx"] Nov 03 01:20:03.996657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.996622 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" Nov 03 01:20:03.996657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.996652 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" Nov 03 01:20:03.996781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:03.996772 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46ed093-2590-4248-ad19-409dd68099a1" containerName="istio-proxy" Nov 03 01:20:04.000040 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.000020 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.054568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.054533 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx"] Nov 03 01:20:04.059915 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.059868 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.060197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.060061 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrph5\" (UniqueName: \"kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.060197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.060157 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.161191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.161147 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrph5\" (UniqueName: \"kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.161357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.161285 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.161877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.161848 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.162696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.162670 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.164106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.164075 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.217356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.217327 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrph5\" (UniqueName: \"kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5\") pod \"maas-api-revoke-sub-6a3112e3-ckbgx\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.237241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.237216 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:04.262517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.262489 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca\") pod \"16362872-a310-4406-8534-117656535136\" (UID: \"16362872-a310-4406-8534-117656535136\") " Nov 03 01:20:04.262675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.262601 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7\" (UniqueName: \"kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7\") pod \"16362872-a310-4406-8534-117656535136\" (UID: \"16362872-a310-4406-8534-117656535136\") " Nov 03 01:20:04.262765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.262751 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca\") pod \"16362872-a310-4406-8534-117656535136\" (UID: \"16362872-a310-4406-8534-117656535136\") " Nov 03 01:20:04.262863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.262839 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca" pod "16362872-a310-4406-8534-117656535136" (UID: "16362872-a310-4406-8534-117656535136"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:04.263106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.263089 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16362872-a310-4406-8534-117656535136-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:04.264800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.264777 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca" pod "16362872-a310-4406-8534-117656535136" (UID: "16362872-a310-4406-8534-117656535136"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:04.264800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.264790 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7" pod "16362872-a310-4406-8534-117656535136" (UID: "16362872-a310-4406-8534-117656535136"). InnerVolumeSpecName "kube-api-access-879l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:04.313171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.313150 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:04.363983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.363955 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-879l7\" (UniqueName: \"kubernetes.io/projected/16362872-a310-4406-8534-117656535136-kube-api-access-879l7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:04.363983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.363983 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16362872-a310-4406-8534-117656535136-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:04.693619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:04.693589 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx"] Nov 03 01:20:04.694073 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:20:04.694046 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e586190_7c6e_47f8_aed5_96bb5607cef9.slice/crio-e60d23dcce268c32ad0f592cd7ad675e6177c87778e47630f98b42bc765b31b5 WatchSource:0}: Error finding container e60d23dcce268c32ad0f592cd7ad675e6177c87778e47630f98b42bc765b31b5: Status 404 returned error can't find the container with id e60d23dcce268c32ad0f592cd7ad675e6177c87778e47630f98b42bc765b31b5 Nov 03 01:20:05.073692 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:05.073657 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" event={"ID":"7e586190-7c6e-47f8-aed5-96bb5607cef9","Type":"ContainerStarted","Data":"c7b71983069ea366dae3f610f4adab9da203b414e3b0a0777f3f500d526b4b8a"} Nov 03 01:20:05.074099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:05.073716 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" event={"ID":"7e586190-7c6e-47f8-aed5-96bb5607cef9","Type":"ContainerStarted","Data":"e60d23dcce268c32ad0f592cd7ad675e6177c87778e47630f98b42bc765b31b5"} Nov 03 01:20:05.075265 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:05.075245 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" Nov 03 01:20:05.075377 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:05.075269 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj" event={"ID":"16362872-a310-4406-8534-117656535136","Type":"ContainerDied","Data":"410bd7c14aa9ae1c079c7cb9265d71182de47c846fcb742301d6262337f780f3"} Nov 03 01:20:05.075377 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:05.075304 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410bd7c14aa9ae1c079c7cb9265d71182de47c846fcb742301d6262337f780f3" Nov 03 01:20:05.118331 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:05.118307 5144 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16362872_a310_4406_8534_117656535136.slice\": RecentStats: unable to find data in memory cache]" Nov 03 01:20:05.189656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:05.189581 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" podStartSLOduration=2.18956756 podStartE2EDuration="2.18956756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:05.187129871 +0000 UTC m=+4495.332824326" watchObservedRunningTime="2026-11-03 01:20:05.18956756 +0000 UTC m=+4495.335262015" Nov 03 01:20:06.083410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:06.083378 5144 generic.go:356] "Generic (PLEG): container finished" podID="7e586190-7c6e-47f8-aed5-96bb5607cef9" containerID="c7b71983069ea366dae3f610f4adab9da203b414e3b0a0777f3f500d526b4b8a" exitCode=0 Nov 03 01:20:06.083865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:06.083465 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" event={"ID":"7e586190-7c6e-47f8-aed5-96bb5607cef9","Type":"ContainerDied","Data":"c7b71983069ea366dae3f610f4adab9da203b414e3b0a0777f3f500d526b4b8a"} Nov 03 01:20:06.818995 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:06.818954 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" podUID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" containerName="main" containerID="cri-o://8a0e6c7f09703fb2b374c8cd379d012a4dc9e765c5d06fe094a8fa252584a987" gracePeriod=45 Nov 03 01:20:06.835922 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:06.835887 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" containerID="cri-o://d4162ce811e0a971af5edb42f87d85795db00327f878a72e66a945afd0ef7d3f" gracePeriod=45 Nov 03 01:20:06.924098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:06.924065 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" probeResult="failure" output="Get \"https://10.132.0.81:8000/ready\": dial tcp 10.132.0.81:8000: connect: connection refused" Nov 03 01:20:07.096978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.096942 5144 generic.go:356] "Generic (PLEG): container finished" podID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerID="d4162ce811e0a971af5edb42f87d85795db00327f878a72e66a945afd0ef7d3f" exitCode=0 Nov 03 01:20:07.097365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.097015 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" event={"ID":"3be973bd-a2eb-4026-a290-2aa35e7e08af","Type":"ContainerDied","Data":"d4162ce811e0a971af5edb42f87d85795db00327f878a72e66a945afd0ef7d3f"} Nov 03 01:20:07.098800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.098776 5144 generic.go:356] "Generic (PLEG): container finished" podID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" containerID="8a0e6c7f09703fb2b374c8cd379d012a4dc9e765c5d06fe094a8fa252584a987" exitCode=0 Nov 03 01:20:07.098941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.098825 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" event={"ID":"b9c2511d-43e0-4a2e-8730-d851ec97e70b","Type":"ContainerDied","Data":"8a0e6c7f09703fb2b374c8cd379d012a4dc9e765c5d06fe094a8fa252584a987"} Nov 03 01:20:07.098941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.098866 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" event={"ID":"b9c2511d-43e0-4a2e-8730-d851ec97e70b","Type":"ContainerDied","Data":"d19c8f6d595acefe2278211f18e8bf9aa645a7b07c1655e33d46a7c57f0febe6"} Nov 03 01:20:07.098941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.098886 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d19c8f6d595acefe2278211f18e8bf9aa645a7b07c1655e33d46a7c57f0febe6" Nov 03 01:20:07.104620 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.104598 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 01:20:07.135965 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.135866 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.190950 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache\") pod \"3be973bd-a2eb-4026-a290-2aa35e7e08af\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191008 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir\") pod \"3be973bd-a2eb-4026-a290-2aa35e7e08af\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191042 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg\" (UniqueName: \"kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg\") pod \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191075 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home\") pod \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191106 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir\") pod \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191145 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs\" (UniqueName: \"kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs\") pod \"3be973bd-a2eb-4026-a290-2aa35e7e08af\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191171 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm\") pod \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191212 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home\") pod \"3be973bd-a2eb-4026-a290-2aa35e7e08af\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191235 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm\") pod \"3be973bd-a2eb-4026-a290-2aa35e7e08af\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191258 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache\") pod \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191292 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj\" (UniqueName: \"kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj\") pod \"3be973bd-a2eb-4026-a290-2aa35e7e08af\" (UID: \"3be973bd-a2eb-4026-a290-2aa35e7e08af\") " Nov 03 01:20:07.191981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.191355 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs\") pod \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\" (UID: \"b9c2511d-43e0-4a2e-8730-d851ec97e70b\") " Nov 03 01:20:07.192500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.192212 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home" pod "b9c2511d-43e0-4a2e-8730-d851ec97e70b" (UID: "b9c2511d-43e0-4a2e-8730-d851ec97e70b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.192927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.192700 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir" pod "3be973bd-a2eb-4026-a290-2aa35e7e08af" (UID: "3be973bd-a2eb-4026-a290-2aa35e7e08af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.192927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.192757 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache" pod "b9c2511d-43e0-4a2e-8730-d851ec97e70b" (UID: "b9c2511d-43e0-4a2e-8730-d851ec97e70b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.193114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.192935 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache" pod "3be973bd-a2eb-4026-a290-2aa35e7e08af" (UID: "3be973bd-a2eb-4026-a290-2aa35e7e08af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.193210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.193172 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir" pod "b9c2511d-43e0-4a2e-8730-d851ec97e70b" (UID: "b9c2511d-43e0-4a2e-8730-d851ec97e70b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.193355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.193229 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home" pod "3be973bd-a2eb-4026-a290-2aa35e7e08af" (UID: "3be973bd-a2eb-4026-a290-2aa35e7e08af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.195561 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.195526 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj" pod "3be973bd-a2eb-4026-a290-2aa35e7e08af" (UID: "3be973bd-a2eb-4026-a290-2aa35e7e08af"). InnerVolumeSpecName "kube-api-access-4ffzj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:07.195941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.195905 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm" pod "b9c2511d-43e0-4a2e-8730-d851ec97e70b" (UID: "b9c2511d-43e0-4a2e-8730-d851ec97e70b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.196778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.196739 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm" pod "3be973bd-a2eb-4026-a290-2aa35e7e08af" (UID: "3be973bd-a2eb-4026-a290-2aa35e7e08af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:20:07.198094 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.198064 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs" pod "3be973bd-a2eb-4026-a290-2aa35e7e08af" (UID: "3be973bd-a2eb-4026-a290-2aa35e7e08af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:07.198194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.198153 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs" pod "b9c2511d-43e0-4a2e-8730-d851ec97e70b" (UID: "b9c2511d-43e0-4a2e-8730-d851ec97e70b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:07.199426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.199402 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg" pod "b9c2511d-43e0-4a2e-8730-d851ec97e70b" (UID: "b9c2511d-43e0-4a2e-8730-d851ec97e70b"). InnerVolumeSpecName "kube-api-access-887tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:07.250559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.250521 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:07.292130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292099 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca\") pod \"7e586190-7c6e-47f8-aed5-96bb5607cef9\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " Nov 03 01:20:07.292330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292150 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5\" (UniqueName: \"kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5\") pod \"7e586190-7c6e-47f8-aed5-96bb5607cef9\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " Nov 03 01:20:07.292330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292213 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca\") pod \"7e586190-7c6e-47f8-aed5-96bb5607cef9\" (UID: \"7e586190-7c6e-47f8-aed5-96bb5607cef9\") " Nov 03 01:20:07.292513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292488 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca" pod "7e586190-7c6e-47f8-aed5-96bb5607cef9" (UID: "7e586190-7c6e-47f8-aed5-96bb5607cef9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:07.292648 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292610 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292652 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292667 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292680 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-887tg\" (UniqueName: \"kubernetes.io/projected/b9c2511d-43e0-4a2e-8730-d851ec97e70b-kube-api-access-887tg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292691 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292702 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292713 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3be973bd-a2eb-4026-a290-2aa35e7e08af-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292725 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292738 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292749 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3be973bd-a2eb-4026-a290-2aa35e7e08af-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292760 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b9c2511d-43e0-4a2e-8730-d851ec97e70b-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292774 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ffzj\" (UniqueName: \"kubernetes.io/projected/3be973bd-a2eb-4026-a290-2aa35e7e08af-kube-api-access-4ffzj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.292977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.292786 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9c2511d-43e0-4a2e-8730-d851ec97e70b-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.294448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.294425 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca" pod "7e586190-7c6e-47f8-aed5-96bb5607cef9" (UID: "7e586190-7c6e-47f8-aed5-96bb5607cef9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:07.294784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.294765 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5" pod "7e586190-7c6e-47f8-aed5-96bb5607cef9" (UID: "7e586190-7c6e-47f8-aed5-96bb5607cef9"). InnerVolumeSpecName "kube-api-access-qrph5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:07.394053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.393974 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrph5\" (UniqueName: \"kubernetes.io/projected/7e586190-7c6e-47f8-aed5-96bb5607cef9-kube-api-access-qrph5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:07.394053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:07.394002 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e586190-7c6e-47f8-aed5-96bb5607cef9-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:08.106900 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.106835 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" Nov 03 01:20:08.106900 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.106858 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn" event={"ID":"3be973bd-a2eb-4026-a290-2aa35e7e08af","Type":"ContainerDied","Data":"f786f2c7450a73946959080fad1f4dd5eadf9881131e39b70313b2966670d57f"} Nov 03 01:20:08.107405 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.106908 5144 scope.go:122] "RemoveContainer" containerID="d4162ce811e0a971af5edb42f87d85795db00327f878a72e66a945afd0ef7d3f" Nov 03 01:20:08.108353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.108335 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" Nov 03 01:20:08.108463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.108396 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx" event={"ID":"7e586190-7c6e-47f8-aed5-96bb5607cef9","Type":"ContainerDied","Data":"e60d23dcce268c32ad0f592cd7ad675e6177c87778e47630f98b42bc765b31b5"} Nov 03 01:20:08.108525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.108472 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e60d23dcce268c32ad0f592cd7ad675e6177c87778e47630f98b42bc765b31b5" Nov 03 01:20:08.108579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.108557 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq" Nov 03 01:20:08.269431 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.269399 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn"] Nov 03 01:20:08.275403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.275376 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/simulator-w2-531ee4-kserve-698bd48d84-8lzbn"] Nov 03 01:20:08.383369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.383297 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq"] Nov 03 01:20:08.386448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.386424 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-531ee4/distinct-w2-531ee4-kserve-589bffb7b8-jrwmq"] Nov 03 01:20:08.516418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.516383 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" containerName="payload-processing" containerID="cri-o://d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db" gracePeriod=25 Nov 03 01:20:08.622222 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:08.622178 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerName="payload-pre-processing" containerID="cri-o://0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9" gracePeriod=25 Nov 03 01:20:09.997101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:09.997073 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" path="/var/lib/kubelet/pods/3be973bd-a2eb-4026-a290-2aa35e7e08af/volumes" Nov 03 01:20:09.997487 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:09.997474 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" path="/var/lib/kubelet/pods/b9c2511d-43e0-4a2e-8730-d851ec97e70b/volumes" Nov 03 01:20:10.799256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:10.799226 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp"] Nov 03 01:20:10.906197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:10.906166 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64"] Nov 03 01:20:11.120687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:11.120592 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:20:11.136211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:11.136187 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:20:12.154578 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:12.154538 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:20:12.940397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940366 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc"] Nov 03 01:20:12.940906 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940893 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7e586190-7c6e-47f8-aed5-96bb5607cef9" containerName="revoke-keys" Nov 03 01:20:12.940955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940908 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e586190-7c6e-47f8-aed5-96bb5607cef9" containerName="revoke-keys" Nov 03 01:20:12.940955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940925 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" Nov 03 01:20:12.940955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940931 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" Nov 03 01:20:12.940955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940937 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" containerName="main" Nov 03 01:20:12.940955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940942 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" containerName="main" Nov 03 01:20:12.941118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940959 5144 cpu_manager.go:414] "containerMap: removing container" podUID="16362872-a310-4406-8534-117656535136" containerName="revoke-keys" Nov 03 01:20:12.941118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.940964 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16362872-a310-4406-8534-117656535136" containerName="revoke-keys" Nov 03 01:20:12.941118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.941048 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="16362872-a310-4406-8534-117656535136" containerName="revoke-keys" Nov 03 01:20:12.941118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.941061 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e586190-7c6e-47f8-aed5-96bb5607cef9" containerName="revoke-keys" Nov 03 01:20:12.941118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.941069 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3be973bd-a2eb-4026-a290-2aa35e7e08af" containerName="main" Nov 03 01:20:12.941118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.941076 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9c2511d-43e0-4a2e-8730-d851ec97e70b" containerName="main" Nov 03 01:20:12.945861 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.945839 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:12.999219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:12.999192 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc"] Nov 03 01:20:13.044798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.044773 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:20:13.044948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.044815 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:20:13.047219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.047198 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7k9l\" (UniqueName: \"kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.047343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.047230 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.047412 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.047376 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.148871 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.148846 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7k9l\" (UniqueName: \"kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.148993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.148879 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.148993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.148939 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.149577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.149560 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.151283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.151266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.205321 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.205270 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7k9l\" (UniqueName: \"kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l\") pod \"maas-api-revoke-sub-b3430b4b-vxxnc\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.259171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.259147 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:13.638319 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:13.638289 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc"] Nov 03 01:20:13.639162 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:20:13.639135 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddd8fce_395f_4776_9988_b67eaa4a24f1.slice/crio-e0c17399fee11eba6ed5d19fa30560817bc114685e8b67de543177dfb07a7f45 WatchSource:0}: Error finding container e0c17399fee11eba6ed5d19fa30560817bc114685e8b67de543177dfb07a7f45: Status 404 returned error can't find the container with id e0c17399fee11eba6ed5d19fa30560817bc114685e8b67de543177dfb07a7f45 Nov 03 01:20:14.022466 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:14.022438 5144 scope.go:122] "RemoveContainer" containerID="8a0e6c7f09703fb2b374c8cd379d012a4dc9e765c5d06fe094a8fa252584a987" Nov 03 01:20:14.152518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:14.152483 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" event={"ID":"0ddd8fce-395f-4776-9988-b67eaa4a24f1","Type":"ContainerStarted","Data":"7daedf6e6dfd7038d97a5f8b1533bca0de101f7dad7814c103df203154ef0b91"} Nov 03 01:20:14.152675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:14.152541 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" event={"ID":"0ddd8fce-395f-4776-9988-b67eaa4a24f1","Type":"ContainerStarted","Data":"e0c17399fee11eba6ed5d19fa30560817bc114685e8b67de543177dfb07a7f45"} Nov 03 01:20:14.267297 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:14.267254 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" podStartSLOduration=2.267241005 podStartE2EDuration="2.267241005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:14.264434459 +0000 UTC m=+4504.410128915" watchObservedRunningTime="2026-11-03 01:20:14.267241005 +0000 UTC m=+4504.412935460" Nov 03 01:20:15.160615 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:15.160579 5144 generic.go:356] "Generic (PLEG): container finished" podID="0ddd8fce-395f-4776-9988-b67eaa4a24f1" containerID="7daedf6e6dfd7038d97a5f8b1533bca0de101f7dad7814c103df203154ef0b91" exitCode=0 Nov 03 01:20:15.160814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:15.160668 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" event={"ID":"0ddd8fce-395f-4776-9988-b67eaa4a24f1","Type":"ContainerDied","Data":"7daedf6e6dfd7038d97a5f8b1533bca0de101f7dad7814c103df203154ef0b91"} Nov 03 01:20:15.344175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:15.344140 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:15.799503 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:15.799464 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" containerName="payload-processing" containerID="cri-o://aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff" gracePeriod=25 Nov 03 01:20:15.907263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:15.907208 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerName="payload-pre-processing" containerID="cri-o://836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b" gracePeriod=25 Nov 03 01:20:16.316113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.316087 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:16.350191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.350155 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:16.382075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.382047 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca\") pod \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " Nov 03 01:20:16.382194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.382101 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l\" (UniqueName: \"kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l\") pod \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " Nov 03 01:20:16.382194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.382139 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca\") pod \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\" (UID: \"0ddd8fce-395f-4776-9988-b67eaa4a24f1\") " Nov 03 01:20:16.382417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.382395 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca" pod "0ddd8fce-395f-4776-9988-b67eaa4a24f1" (UID: "0ddd8fce-395f-4776-9988-b67eaa4a24f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:16.382726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.382702 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:16.384409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.384390 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca" pod "0ddd8fce-395f-4776-9988-b67eaa4a24f1" (UID: "0ddd8fce-395f-4776-9988-b67eaa4a24f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:16.384508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.384490 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l" pod "0ddd8fce-395f-4776-9988-b67eaa4a24f1" (UID: "0ddd8fce-395f-4776-9988-b67eaa4a24f1"). InnerVolumeSpecName "kube-api-access-x7k9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:16.483852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.483777 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7k9l\" (UniqueName: \"kubernetes.io/projected/0ddd8fce-395f-4776-9988-b67eaa4a24f1-kube-api-access-x7k9l\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:16.483852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:16.483805 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddd8fce-395f-4776-9988-b67eaa4a24f1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:17.174105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:17.174076 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" Nov 03 01:20:17.174285 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:17.174116 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc" event={"ID":"0ddd8fce-395f-4776-9988-b67eaa4a24f1","Type":"ContainerDied","Data":"e0c17399fee11eba6ed5d19fa30560817bc114685e8b67de543177dfb07a7f45"} Nov 03 01:20:17.174285 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:17.174167 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0c17399fee11eba6ed5d19fa30560817bc114685e8b67de543177dfb07a7f45" Nov 03 01:20:19.904557 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:19.904517 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:20.917293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:20.917252 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:23.901727 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.901688 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq"] Nov 03 01:20:23.902248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.902233 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0ddd8fce-395f-4776-9988-b67eaa4a24f1" containerName="revoke-keys" Nov 03 01:20:23.902248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.902247 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddd8fce-395f-4776-9988-b67eaa4a24f1" containerName="revoke-keys" Nov 03 01:20:23.902395 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.902328 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddd8fce-395f-4776-9988-b67eaa4a24f1" containerName="revoke-keys" Nov 03 01:20:23.905991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.905964 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:23.961756 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.961726 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq"] Nov 03 01:20:23.979901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.979871 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:23.980038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.980019 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:23.980089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:23.980055 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpqwt\" (UniqueName: \"kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.006491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.006468 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:20:24.006623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.006556 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:20:24.081029 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.080992 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.081221 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.081062 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpqwt\" (UniqueName: \"kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.081909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.081331 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.082120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.082093 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.084261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.084242 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.138228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.138198 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpqwt\" (UniqueName: \"kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt\") pod \"maas-api-revoke-sub-c812d30b-bkxjq\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.219906 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.219827 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:24.625909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:24.625876 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq"] Nov 03 01:20:24.626439 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:20:24.626410 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39f9c304_c69f_45a3_b5d0_2e605438984a.slice/crio-1255b61644d356a92f31fedfee9cc017adb685acfb1b45c59bc096c7c80be573 WatchSource:0}: Error finding container 1255b61644d356a92f31fedfee9cc017adb685acfb1b45c59bc096c7c80be573: Status 404 returned error can't find the container with id 1255b61644d356a92f31fedfee9cc017adb685acfb1b45c59bc096c7c80be573 Nov 03 01:20:25.241624 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:25.241544 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" event={"ID":"39f9c304-c69f-45a3-b5d0-2e605438984a","Type":"ContainerStarted","Data":"4a27b7ab35755a472225333adb6d7753f8278559af85ce1189edb43852932e94"} Nov 03 01:20:25.241624 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:25.241601 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" event={"ID":"39f9c304-c69f-45a3-b5d0-2e605438984a","Type":"ContainerStarted","Data":"1255b61644d356a92f31fedfee9cc017adb685acfb1b45c59bc096c7c80be573"} Nov 03 01:20:25.344664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:25.344617 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:25.355839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:25.355798 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" podStartSLOduration=2.355785861 podStartE2EDuration="2.355785861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:25.352709046 +0000 UTC m=+4515.498403512" watchObservedRunningTime="2026-11-03 01:20:25.355785861 +0000 UTC m=+4515.501480317" Nov 03 01:20:26.249228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:26.249198 5144 generic.go:356] "Generic (PLEG): container finished" podID="39f9c304-c69f-45a3-b5d0-2e605438984a" containerID="4a27b7ab35755a472225333adb6d7753f8278559af85ce1189edb43852932e94" exitCode=0 Nov 03 01:20:26.249618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:26.249254 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" event={"ID":"39f9c304-c69f-45a3-b5d0-2e605438984a","Type":"ContainerDied","Data":"4a27b7ab35755a472225333adb6d7753f8278559af85ce1189edb43852932e94"} Nov 03 01:20:26.350404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:26.350376 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:27.411527 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.411503 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:27.525214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.525050 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt\" (UniqueName: \"kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt\") pod \"39f9c304-c69f-45a3-b5d0-2e605438984a\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " Nov 03 01:20:27.525214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.525199 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca\") pod \"39f9c304-c69f-45a3-b5d0-2e605438984a\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " Nov 03 01:20:27.525438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.525254 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca\") pod \"39f9c304-c69f-45a3-b5d0-2e605438984a\" (UID: \"39f9c304-c69f-45a3-b5d0-2e605438984a\") " Nov 03 01:20:27.526035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.525987 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca" pod "39f9c304-c69f-45a3-b5d0-2e605438984a" (UID: "39f9c304-c69f-45a3-b5d0-2e605438984a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:27.527809 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.527786 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt" pod "39f9c304-c69f-45a3-b5d0-2e605438984a" (UID: "39f9c304-c69f-45a3-b5d0-2e605438984a"). InnerVolumeSpecName "kube-api-access-hpqwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:27.527972 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.527947 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca" pod "39f9c304-c69f-45a3-b5d0-2e605438984a" (UID: "39f9c304-c69f-45a3-b5d0-2e605438984a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:20:27.626667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.626613 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpqwt\" (UniqueName: \"kubernetes.io/projected/39f9c304-c69f-45a3-b5d0-2e605438984a-kube-api-access-hpqwt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:27.626667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.626670 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:27.626831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:27.626683 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39f9c304-c69f-45a3-b5d0-2e605438984a-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:28.264645 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.264607 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" Nov 03 01:20:28.264830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.264658 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq" event={"ID":"39f9c304-c69f-45a3-b5d0-2e605438984a","Type":"ContainerDied","Data":"1255b61644d356a92f31fedfee9cc017adb685acfb1b45c59bc096c7c80be573"} Nov 03 01:20:28.264830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.264703 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1255b61644d356a92f31fedfee9cc017adb685acfb1b45c59bc096c7c80be573" Nov 03 01:20:28.820780 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.818660 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 01:20:28.931268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.931237 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 01:20:28.939030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.938998 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf\" (UniqueName: \"kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf\") pod \"da759ad9-314a-43b7-ab7d-597df96470cc\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " Nov 03 01:20:28.939166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.939125 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume\") pod \"da759ad9-314a-43b7-ab7d-597df96470cc\" (UID: \"da759ad9-314a-43b7-ab7d-597df96470cc\") " Nov 03 01:20:28.939470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.939451 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume" pod "da759ad9-314a-43b7-ab7d-597df96470cc" (UID: "da759ad9-314a-43b7-ab7d-597df96470cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:28.941469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:28.941439 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf" pod "da759ad9-314a-43b7-ab7d-597df96470cc" (UID: "da759ad9-314a-43b7-ab7d-597df96470cc"). InnerVolumeSpecName "kube-api-access-ztfpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:29.040408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.040333 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f\" (UniqueName: \"kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f\") pod \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " Nov 03 01:20:29.040557 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.040457 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume\") pod \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\" (UID: \"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0\") " Nov 03 01:20:29.040690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.040676 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da759ad9-314a-43b7-ab7d-597df96470cc-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:29.040748 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.040692 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztfpf\" (UniqueName: \"kubernetes.io/projected/da759ad9-314a-43b7-ab7d-597df96470cc-kube-api-access-ztfpf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:29.040845 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.040825 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume" pod "d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" (UID: "d6a2f16b-0107-4a03-94c4-1a5abe76b7a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:29.042596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.042577 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f" pod "d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" (UID: "d6a2f16b-0107-4a03-94c4-1a5abe76b7a0"). InnerVolumeSpecName "kube-api-access-szg2f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:29.141757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.141719 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szg2f\" (UniqueName: \"kubernetes.io/projected/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-kube-api-access-szg2f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:29.141757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.141752 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:29.273332 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.273302 5144 generic.go:356] "Generic (PLEG): container finished" podID="da759ad9-314a-43b7-ab7d-597df96470cc" containerID="d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db" exitCode=0 Nov 03 01:20:29.273520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.273373 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" Nov 03 01:20:29.273520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.273396 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" event={"ID":"da759ad9-314a-43b7-ab7d-597df96470cc","Type":"ContainerDied","Data":"d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db"} Nov 03 01:20:29.273520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.273452 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn" event={"ID":"da759ad9-314a-43b7-ab7d-597df96470cc","Type":"ContainerDied","Data":"f696a1cc671c6d07454e60a087554c939738740362fe907edb6433783206389a"} Nov 03 01:20:29.273520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.273485 5144 scope.go:122] "RemoveContainer" containerID="d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db" Nov 03 01:20:29.275272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.275247 5144 generic.go:356] "Generic (PLEG): container finished" podID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerID="0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9" exitCode=0 Nov 03 01:20:29.275436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.275345 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" event={"ID":"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0","Type":"ContainerDied","Data":"0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9"} Nov 03 01:20:29.275436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.275354 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" Nov 03 01:20:29.275436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.275381 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn" event={"ID":"d6a2f16b-0107-4a03-94c4-1a5abe76b7a0","Type":"ContainerDied","Data":"2cc48b6fec2048b2639b0e06df3d8697011c6a32a0ba7153e8a3c50846469544"} Nov 03 01:20:29.286224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.286210 5144 scope.go:122] "RemoveContainer" containerID="d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db" Nov 03 01:20:29.286503 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:29.286481 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db\": container with ID starting with d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db not found: ID does not exist" containerID="d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db" Nov 03 01:20:29.286570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.286512 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db"} err="failed to get container status \"d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db\": rpc error: code = NotFound desc = could not find container \"d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db\": container with ID starting with d4398c392a3ad37ae864ed9a8c741168e3ef1386f720c5cde69dd4a5e88bc2db not found: ID does not exist" Nov 03 01:20:29.286570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.286527 5144 scope.go:122] "RemoveContainer" containerID="0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9" Nov 03 01:20:29.296172 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.296155 5144 scope.go:122] "RemoveContainer" containerID="0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9" Nov 03 01:20:29.296457 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:29.296432 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9\": container with ID starting with 0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9 not found: ID does not exist" containerID="0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9" Nov 03 01:20:29.296549 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.296456 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9"} err="failed to get container status \"0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9\": rpc error: code = NotFound desc = could not find container \"0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9\": container with ID starting with 0a94e990ffa076a6c78d38d4a1dc07b4d81a53d607fd3a501c0cf78c12c6e1a9 not found: ID does not exist" Nov 03 01:20:29.436572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.436529 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn"] Nov 03 01:20:29.485816 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.485792 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-531ee4-696c6dfb7c-pkwjn"] Nov 03 01:20:29.547369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.547303 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn"] Nov 03 01:20:29.551158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.551136 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-531ee4-5964cc6c8c-j44sn"] Nov 03 01:20:29.904160 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.904073 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:29.997731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.997702 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" path="/var/lib/kubelet/pods/d6a2f16b-0107-4a03-94c4-1a5abe76b7a0/volumes" Nov 03 01:20:29.998057 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:29.998044 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" path="/var/lib/kubelet/pods/da759ad9-314a-43b7-ab7d-597df96470cc/volumes" Nov 03 01:20:30.917117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:30.917082 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:20:36.084127 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.084104 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 01:20:36.104162 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.104116 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2\" (UniqueName: \"kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2\") pod \"e96a426d-4339-4402-a7b4-fd6973106a08\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " Nov 03 01:20:36.104162 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.104162 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume\") pod \"e96a426d-4339-4402-a7b4-fd6973106a08\" (UID: \"e96a426d-4339-4402-a7b4-fd6973106a08\") " Nov 03 01:20:36.104618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.104588 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume" pod "e96a426d-4339-4402-a7b4-fd6973106a08" (UID: "e96a426d-4339-4402-a7b4-fd6973106a08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:36.106608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.106580 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2" pod "e96a426d-4339-4402-a7b4-fd6973106a08" (UID: "e96a426d-4339-4402-a7b4-fd6973106a08"). InnerVolumeSpecName "kube-api-access-9kqb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:36.166666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.166608 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 01:20:36.205104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.205073 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume\") pod \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " Nov 03 01:20:36.205234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.205109 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9\" (UniqueName: \"kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9\") pod \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\" (UID: \"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac\") " Nov 03 01:20:36.205401 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.205382 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kqb2\" (UniqueName: \"kubernetes.io/projected/e96a426d-4339-4402-a7b4-fd6973106a08-kube-api-access-9kqb2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.205444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.205407 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e96a426d-4339-4402-a7b4-fd6973106a08-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.205520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.205493 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume" pod "1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" (UID: "1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:20:36.207826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.207798 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9" pod "1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" (UID: "1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac"). InnerVolumeSpecName "kube-api-access-2p8p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:20:36.305928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.305894 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.305928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.305923 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p8p9\" (UniqueName: \"kubernetes.io/projected/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac-kube-api-access-2p8p9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:20:36.335071 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.334991 5144 generic.go:356] "Generic (PLEG): container finished" podID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerID="836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b" exitCode=0 Nov 03 01:20:36.335071 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.335058 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" Nov 03 01:20:36.335294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.335091 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" event={"ID":"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac","Type":"ContainerDied","Data":"836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b"} Nov 03 01:20:36.335294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.335141 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64" event={"ID":"1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac","Type":"ContainerDied","Data":"4578bb6b41a86f4fdb5bbc873a54bd69ae572ac77a407eb018e50f1364c0f3f9"} Nov 03 01:20:36.335294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.335169 5144 scope.go:122] "RemoveContainer" containerID="836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b" Nov 03 01:20:36.337738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.336927 5144 generic.go:356] "Generic (PLEG): container finished" podID="e96a426d-4339-4402-a7b4-fd6973106a08" containerID="aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff" exitCode=0 Nov 03 01:20:36.337738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.336961 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" event={"ID":"e96a426d-4339-4402-a7b4-fd6973106a08","Type":"ContainerDied","Data":"aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff"} Nov 03 01:20:36.337738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.336984 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" Nov 03 01:20:36.337738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.337020 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp" event={"ID":"e96a426d-4339-4402-a7b4-fd6973106a08","Type":"ContainerDied","Data":"d80eb0dc77440b6ec4d8445ebf2cf4763dd6091d9e7c577487d8b2e3fa7dab41"} Nov 03 01:20:36.347837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.347476 5144 scope.go:122] "RemoveContainer" containerID="836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b" Nov 03 01:20:36.348076 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:36.347992 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b\": container with ID starting with 836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b not found: ID does not exist" containerID="836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b" Nov 03 01:20:36.348076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.348026 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b"} err="failed to get container status \"836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b\": rpc error: code = NotFound desc = could not find container \"836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b\": container with ID starting with 836cd642d3aecf910047ca9e0d45c06afa82ff6fa171ef8de69eee763edbdc5b not found: ID does not exist" Nov 03 01:20:36.348076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.348049 5144 scope.go:122] "RemoveContainer" containerID="aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff" Nov 03 01:20:36.357734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.357713 5144 scope.go:122] "RemoveContainer" containerID="aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff" Nov 03 01:20:36.358017 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:36.357999 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff\": container with ID starting with aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff not found: ID does not exist" containerID="aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff" Nov 03 01:20:36.358082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.358023 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff"} err="failed to get container status \"aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff\": rpc error: code = NotFound desc = could not find container \"aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff\": container with ID starting with aab7cdc3a94fecc586e271ba5cfa0765ae956bebb1f63c52dbb49802cbd26aff not found: ID does not exist" Nov 03 01:20:36.499269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.499229 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp"] Nov 03 01:20:36.502399 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.502375 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-fce94b-66f476d5bc-2stmp"] Nov 03 01:20:36.610830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.610761 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64"] Nov 03 01:20:36.614966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:36.614938 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-fce94b-655c4bd695-5gz64"] Nov 03 01:20:37.996104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:37.996072 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" path="/var/lib/kubelet/pods/1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac/volumes" Nov 03 01:20:37.996616 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:37.996599 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" path="/var/lib/kubelet/pods/e96a426d-4339-4402-a7b4-fd6973106a08/volumes" Nov 03 01:20:42.155596 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:20:42.155547 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:20:49.440765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.440672 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl"] Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441242 5144 cpu_manager.go:414] "containerMap: removing container" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" containerName="payload-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441256 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" containerName="payload-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441270 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerName="payload-pre-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441275 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerName="payload-pre-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441287 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerName="payload-pre-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441292 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerName="payload-pre-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441316 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" containerName="payload-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441323 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" containerName="payload-processing" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441330 5144 cpu_manager.go:414] "containerMap: removing container" podUID="39f9c304-c69f-45a3-b5d0-2e605438984a" containerName="revoke-keys" Nov 03 01:20:49.441346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441335 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39f9c304-c69f-45a3-b5d0-2e605438984a" containerName="revoke-keys" Nov 03 01:20:49.441967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441422 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e96a426d-4339-4402-a7b4-fd6973106a08" containerName="payload-processing" Nov 03 01:20:49.441967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441431 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="39f9c304-c69f-45a3-b5d0-2e605438984a" containerName="revoke-keys" Nov 03 01:20:49.441967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441442 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="da759ad9-314a-43b7-ab7d-597df96470cc" containerName="payload-processing" Nov 03 01:20:49.441967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441448 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a2f16b-0107-4a03-94c4-1a5abe76b7a0" containerName="payload-pre-processing" Nov 03 01:20:49.441967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.441455 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d2f3e00-3bd4-4ab8-b7cc-79daded8a8ac" containerName="payload-pre-processing" Nov 03 01:20:49.444864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.444843 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.496758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.496739 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-531ee4-openshift-default-dockercfg-pcjd8\"" Nov 03 01:20:49.530791 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.530767 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.530899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.530801 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.530899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.530883 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.530990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.530920 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.530990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.530941 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.531073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.531038 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.531123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.531080 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.531167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.531140 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.531247 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.531174 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rltnx\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.600563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.600534 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl"] Nov 03 01:20:49.632072 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632045 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632086 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632121 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632142 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rltnx\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632161 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632179 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632230 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632285 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632325 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632709 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632543 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632900 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632878 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632879 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.632977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.632912 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.633272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.633247 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.634598 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.634577 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.634925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.634904 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.688688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.688669 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.688933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.688913 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rltnx\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx\") pod \"e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:49.758178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:49.758156 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:50.210094 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:20:50.210062 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bdcb247_a812_411b_8aa7_0f699213823f.slice/crio-95546b41a6b078fc5a58a1365cb963cb3227b6945fa3c78a7c5a3f405dfb324a WatchSource:0}: Error finding container 95546b41a6b078fc5a58a1365cb963cb3227b6945fa3c78a7c5a3f405dfb324a: Status 404 returned error can't find the container with id 95546b41a6b078fc5a58a1365cb963cb3227b6945fa3c78a7c5a3f405dfb324a Nov 03 01:20:50.211004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.210978 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl"] Nov 03 01:20:50.212241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.212216 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:20:50.212295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.212272 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:20:50.212331 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.212305 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:20:50.440997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.440959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" event={"ID":"4bdcb247-a812-411b-8aa7-0f699213823f","Type":"ContainerStarted","Data":"21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814"} Nov 03 01:20:50.441397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.441044 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" event={"ID":"4bdcb247-a812-411b-8aa7-0f699213823f","Type":"ContainerStarted","Data":"95546b41a6b078fc5a58a1365cb963cb3227b6945fa3c78a7c5a3f405dfb324a"} Nov 03 01:20:50.560778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.560727 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podStartSLOduration=1.5607140290000001 podStartE2EDuration="1.560714029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:50.557495263 +0000 UTC m=+4540.703189718" watchObservedRunningTime="2026-11-03 01:20:50.560714029 +0000 UTC m=+4540.706408541" Nov 03 01:20:50.758739 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:50.758705 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:51.760080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:51.760049 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 03 01:20:51.760458 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:51.760099 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 03 01:20:52.759468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:52.759434 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 03 01:20:52.759653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:52.759496 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 03 01:20:53.045452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:53.045360 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb"] Nov 03 01:20:53.048548 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:53.048521 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56536685-vjmzb"] Nov 03 01:20:53.758875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:53.758832 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 03 01:20:53.759084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:53.758888 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 03 01:20:53.995479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:53.995448 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5225514-5955-43cb-9550-f4ef20aee65e" path="/var/lib/kubelet/pods/e5225514-5955-43cb-9550-f4ef20aee65e/volumes" Nov 03 01:20:54.758917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:54.758883 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 03 01:20:54.759403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:54.758945 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 03 01:20:55.758612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:55.758574 5144 patch_prober.go:29] interesting pod/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" start-of-body= Nov 03 01:20:55.758854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:55.758659 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.95:15021/healthz/ready\": dial tcp 10.132.1.95:15021: connect: connection refused" Nov 03 01:20:56.763351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:56.763324 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:56.763864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:56.763649 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:20:56.764298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:20:56.764282 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:21:01.018484 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:01.018443 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl"] Nov 03 01:21:01.018920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:01.018625 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" containerID="cri-o://21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814" gracePeriod=30 Nov 03 01:21:06.273925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.273893 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:21:06.419427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419349 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419478 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419510 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419558 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419588 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419594 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.419805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419694 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419729 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.419805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419792 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.420060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419800 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.420060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419859 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token\") pod \"4bdcb247-a812-411b-8aa7-0f699213823f\" (UID: \"4bdcb247-a812-411b-8aa7-0f699213823f\") " Nov 03 01:21:06.420060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.419930 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:06.420060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420000 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.420297 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420135 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.420357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420295 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.420357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420317 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bdcb247-a812-411b-8aa7-0f699213823f-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.420357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420332 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.420357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420345 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.420537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.420360 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.421931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.421909 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:06.422499 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.422482 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:21:06.422572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.422546 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "kube-api-access-rltnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:06.422668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.422646 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token" pod "4bdcb247-a812-411b-8aa7-0f699213823f" (UID: "4bdcb247-a812-411b-8aa7-0f699213823f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:06.521389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.521354 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rltnx\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-kube-api-access-rltnx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.521389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.521384 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bdcb247-a812-411b-8aa7-0f699213823f-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.521389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.521393 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bdcb247-a812-411b-8aa7-0f699213823f-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.521696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.521403 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bdcb247-a812-411b-8aa7-0f699213823f-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:06.552782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.552753 5144 generic.go:356] "Generic (PLEG): container finished" podID="4bdcb247-a812-411b-8aa7-0f699213823f" containerID="21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814" exitCode=0 Nov 03 01:21:06.553016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.552806 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" Nov 03 01:21:06.553016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.552847 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" event={"ID":"4bdcb247-a812-411b-8aa7-0f699213823f","Type":"ContainerDied","Data":"21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814"} Nov 03 01:21:06.553016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.552911 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl" event={"ID":"4bdcb247-a812-411b-8aa7-0f699213823f","Type":"ContainerDied","Data":"95546b41a6b078fc5a58a1365cb963cb3227b6945fa3c78a7c5a3f405dfb324a"} Nov 03 01:21:06.553016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.552941 5144 scope.go:122] "RemoveContainer" containerID="21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814" Nov 03 01:21:06.563920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.563899 5144 scope.go:122] "RemoveContainer" containerID="21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814" Nov 03 01:21:06.564187 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:06.564168 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814\": container with ID starting with 21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814 not found: ID does not exist" containerID="21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814" Nov 03 01:21:06.564254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.564199 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814"} err="failed to get container status \"21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814\": rpc error: code = NotFound desc = could not find container \"21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814\": container with ID starting with 21af92623ae77d512b34db3f30365483b41bfad0c785bd5ed1f7e0a144783814 not found: ID does not exist" Nov 03 01:21:06.716517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.716448 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl"] Nov 03 01:21:06.720242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:06.720222 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-531ee4-openshift-default-69d949ddc9-wdhpl"] Nov 03 01:21:07.996452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:07.996418 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" path="/var/lib/kubelet/pods/4bdcb247-a812-411b-8aa7-0f699213823f/volumes" Nov 03 01:21:10.046184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:10.046154 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm"] Nov 03 01:21:10.095708 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:10.095679 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-29661110-5pwnm"] Nov 03 01:21:11.996080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:11.996047 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2113b172-36c2-4b09-8a58-7b68ac100390" path="/var/lib/kubelet/pods/2113b172-36c2-4b09-8a58-7b68ac100390/volumes" Nov 03 01:21:12.154208 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:12.154177 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:21:12.168843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:12.168819 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp"] Nov 03 01:21:12.278620 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:12.278591 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk"] Nov 03 01:21:14.506327 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.506295 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf"] Nov 03 01:21:14.506843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.506828 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" Nov 03 01:21:14.506888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.506844 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" Nov 03 01:21:14.506949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.506940 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bdcb247-a812-411b-8aa7-0f699213823f" containerName="istio-proxy" Nov 03 01:21:14.511911 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.511896 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.566659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.566612 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf"] Nov 03 01:21:14.599033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.599006 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.599174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.599056 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvhgf\" (UniqueName: \"kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.599246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.599199 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.611738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.611712 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:21:14.611849 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.611724 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:21:14.699997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.699973 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.700146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.700031 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.700146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.700095 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvhgf\" (UniqueName: \"kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.700623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.700594 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.702746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.702723 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.757639 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.757567 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvhgf\" (UniqueName: \"kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf\") pod \"maas-api-revoke-sub-70f96db4-65dbf\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:14.824790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:14.824762 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:15.204596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:15.204568 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf"] Nov 03 01:21:15.205522 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:21:15.205495 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3f4cae_8cdf_4626_bff4_b6af597531b4.slice/crio-cbfd5c17e587beaf147f8c5ea6caeda1d877bf44584cd6f450b8cac434612f76 WatchSource:0}: Error finding container cbfd5c17e587beaf147f8c5ea6caeda1d877bf44584cd6f450b8cac434612f76: Status 404 returned error can't find the container with id cbfd5c17e587beaf147f8c5ea6caeda1d877bf44584cd6f450b8cac434612f76 Nov 03 01:21:15.635015 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:15.634978 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" event={"ID":"9a3f4cae-8cdf-4626-bff4-b6af597531b4","Type":"ContainerStarted","Data":"b2cfb1dba2e4b5bb504c4ea2b3e6939001c62cb332d76172911f38182ef8674f"} Nov 03 01:21:15.635467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:15.635037 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" event={"ID":"9a3f4cae-8cdf-4626-bff4-b6af597531b4","Type":"ContainerStarted","Data":"cbfd5c17e587beaf147f8c5ea6caeda1d877bf44584cd6f450b8cac434612f76"} Nov 03 01:21:15.749580 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:15.749531 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" podStartSLOduration=1.749516999 podStartE2EDuration="1.749516999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:15.746478064 +0000 UTC m=+4565.892172521" watchObservedRunningTime="2026-11-03 01:21:15.749516999 +0000 UTC m=+4565.895211454" Nov 03 01:21:16.648582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:16.648550 5144 generic.go:356] "Generic (PLEG): container finished" podID="9a3f4cae-8cdf-4626-bff4-b6af597531b4" containerID="b2cfb1dba2e4b5bb504c4ea2b3e6939001c62cb332d76172911f38182ef8674f" exitCode=0 Nov 03 01:21:16.649005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:16.648653 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" event={"ID":"9a3f4cae-8cdf-4626-bff4-b6af597531b4","Type":"ContainerDied","Data":"b2cfb1dba2e4b5bb504c4ea2b3e6939001c62cb332d76172911f38182ef8674f"} Nov 03 01:21:17.169599 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.169560 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerName="payload-processing" containerID="cri-o://b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45" gracePeriod=25 Nov 03 01:21:17.612708 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.612663 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerName="payload-pre-processing" containerID="cri-o://b58dc8193beaf3222ca8e249e8e65cb04ba60ac1e8c013cc5bde0651822b681c" gracePeriod=25 Nov 03 01:21:17.799026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.799001 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:17.930108 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.930032 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca\") pod \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " Nov 03 01:21:17.930108 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.930085 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf\" (UniqueName: \"kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf\") pod \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " Nov 03 01:21:17.930354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.930160 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca\") pod \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\" (UID: \"9a3f4cae-8cdf-4626-bff4-b6af597531b4\") " Nov 03 01:21:17.930486 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.930460 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca" pod "9a3f4cae-8cdf-4626-bff4-b6af597531b4" (UID: "9a3f4cae-8cdf-4626-bff4-b6af597531b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:17.930855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.930824 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:17.932959 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.932935 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf" pod "9a3f4cae-8cdf-4626-bff4-b6af597531b4" (UID: "9a3f4cae-8cdf-4626-bff4-b6af597531b4"). InnerVolumeSpecName "kube-api-access-qvhgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:17.936039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:17.936006 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca" pod "9a3f4cae-8cdf-4626-bff4-b6af597531b4" (UID: "9a3f4cae-8cdf-4626-bff4-b6af597531b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:18.032434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:18.032406 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvhgf\" (UniqueName: \"kubernetes.io/projected/9a3f4cae-8cdf-4626-bff4-b6af597531b4-kube-api-access-qvhgf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.032536 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:18.032439 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a3f4cae-8cdf-4626-bff4-b6af597531b4-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:18.663619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:18.663586 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" Nov 03 01:21:18.663789 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:18.663616 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf" event={"ID":"9a3f4cae-8cdf-4626-bff4-b6af597531b4","Type":"ContainerDied","Data":"cbfd5c17e587beaf147f8c5ea6caeda1d877bf44584cd6f450b8cac434612f76"} Nov 03 01:21:18.663789 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:18.663691 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cbfd5c17e587beaf147f8c5ea6caeda1d877bf44584cd6f450b8cac434612f76" Nov 03 01:21:18.801066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:18.801037 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z"] Nov 03 01:21:24.419269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:24.419236 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:24.419269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:24.419259 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:30.256514 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.256476 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67"] Nov 03 01:21:30.257053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.257038 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9a3f4cae-8cdf-4626-bff4-b6af597531b4" containerName="revoke-keys" Nov 03 01:21:30.257103 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.257054 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a3f4cae-8cdf-4626-bff4-b6af597531b4" containerName="revoke-keys" Nov 03 01:21:30.257158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.257149 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a3f4cae-8cdf-4626-bff4-b6af597531b4" containerName="revoke-keys" Nov 03 01:21:30.260564 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.260547 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.312319 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.312291 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:21:30.315992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.315971 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67"] Nov 03 01:21:30.340040 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.340010 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.340187 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.340065 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.340361 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.340332 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cl8vj\" (UniqueName: \"kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.361167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.361145 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:21:30.441443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.441410 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.441566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.441511 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cl8vj\" (UniqueName: \"kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.441566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.441554 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.442239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.442167 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.443962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.443940 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.497490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.497465 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cl8vj\" (UniqueName: \"kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj\") pod \"maas-api-revoke-sub-0472eb8b-wfz67\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.573344 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.573289 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:30.946090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:30.946061 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67"] Nov 03 01:21:30.947230 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:21:30.947190 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecbe4fb3_bff4_4f85_9a2e_8fa5c0a8db1c.slice/crio-b2a445f5c2578498a9c39d3edbeea875772e4965d91c7626b722f3128bd2aabe WatchSource:0}: Error finding container b2a445f5c2578498a9c39d3edbeea875772e4965d91c7626b722f3128bd2aabe: Status 404 returned error can't find the container with id b2a445f5c2578498a9c39d3edbeea875772e4965d91c7626b722f3128bd2aabe Nov 03 01:21:31.505280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:31.505254 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf"] Nov 03 01:21:31.613925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:31.613889 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw"] Nov 03 01:21:31.752513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:31.752480 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" event={"ID":"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c","Type":"ContainerStarted","Data":"7a8548c2968ffb32b0aa492235c93ce7f6e4d833565b5d9573c464e029bd1a1f"} Nov 03 01:21:31.752731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:31.752533 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" event={"ID":"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c","Type":"ContainerStarted","Data":"b2a445f5c2578498a9c39d3edbeea875772e4965d91c7626b722f3128bd2aabe"} Nov 03 01:21:31.866214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:31.866104 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" podStartSLOduration=1.8660849430000002 podStartE2EDuration="1.866084943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:21:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:21:31.864803122 +0000 UTC m=+4582.010497591" watchObservedRunningTime="2026-11-03 01:21:31.866084943 +0000 UTC m=+4582.011779401" Nov 03 01:21:32.760287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:32.760255 5144 generic.go:356] "Generic (PLEG): container finished" podID="ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" containerID="7a8548c2968ffb32b0aa492235c93ce7f6e4d833565b5d9573c464e029bd1a1f" exitCode=0 Nov 03 01:21:32.760721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:32.760336 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" event={"ID":"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c","Type":"ContainerDied","Data":"7a8548c2968ffb32b0aa492235c93ce7f6e4d833565b5d9573c464e029bd1a1f"} Nov 03 01:21:33.451499 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.451463 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7"] Nov 03 01:21:33.451783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.451754 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" containerID="cri-o://c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8" gracePeriod=30 Nov 03 01:21:33.822483 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.822415 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" containerID="cri-o://143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199" gracePeriod=45 Nov 03 01:21:33.952764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.952743 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:33.978173 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.978146 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca\") pod \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " Nov 03 01:21:33.978339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.978202 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca\") pod \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " Nov 03 01:21:33.978339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.978274 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj\" (UniqueName: \"kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj\") pod \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\" (UID: \"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c\") " Nov 03 01:21:33.978525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.978502 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca" pod "ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" (UID: "ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:33.978700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.978680 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:33.980363 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.980300 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca" pod "ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" (UID: "ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:33.980920 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:33.980896 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj" pod "ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" (UID: "ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c"). InnerVolumeSpecName "kube-api-access-cl8vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:34.079447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.079406 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.079447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.079441 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cl8vj\" (UniqueName: \"kubernetes.io/projected/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c-kube-api-access-cl8vj\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.084521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.084492 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 01:21:34.170908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.170875 5144 patch_prober.go:29] interesting pod/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:21:34.171078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.170928 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:21:34.180337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180308 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache\") pod \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " Nov 03 01:21:34.180483 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180415 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home\") pod \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " Nov 03 01:21:34.180483 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180450 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx\" (UniqueName: \"kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx\") pod \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " Nov 03 01:21:34.180576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180482 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm\") pod \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " Nov 03 01:21:34.180576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180520 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache" pod "32861b2c-92a4-4763-83eb-9b8e0eb3e083" (UID: "32861b2c-92a4-4763-83eb-9b8e0eb3e083"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.180576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180524 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs\" (UniqueName: \"kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs\") pod \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " Nov 03 01:21:34.180743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180663 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir\") pod \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\" (UID: \"32861b2c-92a4-4763-83eb-9b8e0eb3e083\") " Nov 03 01:21:34.180743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180708 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home" pod "32861b2c-92a4-4763-83eb-9b8e0eb3e083" (UID: "32861b2c-92a4-4763-83eb-9b8e0eb3e083"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.180967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.180945 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir" pod "32861b2c-92a4-4763-83eb-9b8e0eb3e083" (UID: "32861b2c-92a4-4763-83eb-9b8e0eb3e083"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.181105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.181084 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.181185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.181112 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.181185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.181127 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.182903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.182884 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm" pod "32861b2c-92a4-4763-83eb-9b8e0eb3e083" (UID: "32861b2c-92a4-4763-83eb-9b8e0eb3e083"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:34.182974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.182950 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx" pod "32861b2c-92a4-4763-83eb-9b8e0eb3e083" (UID: "32861b2c-92a4-4763-83eb-9b8e0eb3e083"). InnerVolumeSpecName "kube-api-access-2btbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:34.183017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.182991 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs" pod "32861b2c-92a4-4763-83eb-9b8e0eb3e083" (UID: "32861b2c-92a4-4763-83eb-9b8e0eb3e083"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:21:34.282546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.282464 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2btbx\" (UniqueName: \"kubernetes.io/projected/32861b2c-92a4-4763-83eb-9b8e0eb3e083-kube-api-access-2btbx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.282546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.282495 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32861b2c-92a4-4763-83eb-9b8e0eb3e083-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.282546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.282508 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32861b2c-92a4-4763-83eb-9b8e0eb3e083-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:34.419092 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.419056 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:34.419287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.419262 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:34.775410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.775376 5144 generic.go:356] "Generic (PLEG): container finished" podID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerID="143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199" exitCode=0 Nov 03 01:21:34.775558 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.775436 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" Nov 03 01:21:34.775558 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.775461 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" event={"ID":"32861b2c-92a4-4763-83eb-9b8e0eb3e083","Type":"ContainerDied","Data":"143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199"} Nov 03 01:21:34.775558 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.775513 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" event={"ID":"32861b2c-92a4-4763-83eb-9b8e0eb3e083","Type":"ContainerDied","Data":"333e8f727df7339dde8dcf719e0065a49b1d72de1014c1eaaaf41f5de15ae80a"} Nov 03 01:21:34.775558 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.775534 5144 scope.go:122] "RemoveContainer" containerID="143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199" Nov 03 01:21:34.777300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.777267 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" event={"ID":"ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c","Type":"ContainerDied","Data":"b2a445f5c2578498a9c39d3edbeea875772e4965d91c7626b722f3128bd2aabe"} Nov 03 01:21:34.777462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.777336 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2a445f5c2578498a9c39d3edbeea875772e4965d91c7626b722f3128bd2aabe" Nov 03 01:21:34.777462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.777277 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67" Nov 03 01:21:34.786255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.786234 5144 scope.go:122] "RemoveContainer" containerID="143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199" Nov 03 01:21:34.786540 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:34.786515 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199\": container with ID starting with 143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199 not found: ID does not exist" containerID="143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199" Nov 03 01:21:34.786586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.786551 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199"} err="failed to get container status \"143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199\": rpc error: code = NotFound desc = could not find container \"143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199\": container with ID starting with 143f024d24e9c3b9def176e6733cba1f71bd63e9eb37484e38ef890b77dc2199 not found: ID does not exist" Nov 03 01:21:34.937968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.937949 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z"] Nov 03 01:21:34.940962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:34.940944 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z"] Nov 03 01:21:35.009023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:35.009000 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-048135/premium-w4-048135-kserve-6c67f9c5c9-8qs8z" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" probeResult="failure" output="Get \"https://10.132.0.82:8000/ready\": context deadline exceeded" Nov 03 01:21:35.996315 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:35.996280 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" path="/var/lib/kubelet/pods/32861b2c-92a4-4763-83eb-9b8e0eb3e083/volumes" Nov 03 01:21:36.506228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:36.506187 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerName="payload-processing" containerID="cri-o://05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0" gracePeriod=25 Nov 03 01:21:36.614721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:36.614686 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerName="payload-pre-processing" containerID="cri-o://f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0" gracePeriod=25 Nov 03 01:21:37.423684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.423660 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 01:21:37.510218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.510192 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb\") pod \"2a8e5d31-1e1c-4550-b520-33160ad876db\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " Nov 03 01:21:37.510354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.510343 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume\") pod \"2a8e5d31-1e1c-4550-b520-33160ad876db\" (UID: \"2a8e5d31-1e1c-4550-b520-33160ad876db\") " Nov 03 01:21:37.510700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.510679 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume" pod "2a8e5d31-1e1c-4550-b520-33160ad876db" (UID: "2a8e5d31-1e1c-4550-b520-33160ad876db"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:37.512299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.512273 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb" pod "2a8e5d31-1e1c-4550-b520-33160ad876db" (UID: "2a8e5d31-1e1c-4550-b520-33160ad876db"). InnerVolumeSpecName "kube-api-access-dmdsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:37.611338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.611312 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a8e5d31-1e1c-4550-b520-33160ad876db-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:37.611338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.611334 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmdsb\" (UniqueName: \"kubernetes.io/projected/2a8e5d31-1e1c-4550-b520-33160ad876db-kube-api-access-dmdsb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:37.804428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.804390 5144 generic.go:356] "Generic (PLEG): container finished" podID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerID="b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45" exitCode=0 Nov 03 01:21:37.804593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.804438 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" Nov 03 01:21:37.804593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.804471 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" event={"ID":"2a8e5d31-1e1c-4550-b520-33160ad876db","Type":"ContainerDied","Data":"b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45"} Nov 03 01:21:37.804593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.804525 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp" event={"ID":"2a8e5d31-1e1c-4550-b520-33160ad876db","Type":"ContainerDied","Data":"2efe41e663a66324407fc41352effa172d3bdacad9c00e083cbcf757c18e8b84"} Nov 03 01:21:37.804593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.804553 5144 scope.go:122] "RemoveContainer" containerID="b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45" Nov 03 01:21:37.810039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.810011 5144 generic.go:356] "Generic (PLEG): container finished" podID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerID="b58dc8193beaf3222ca8e249e8e65cb04ba60ac1e8c013cc5bde0651822b681c" exitCode=0 Nov 03 01:21:37.810210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.810180 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" event={"ID":"ee41c01d-7380-4c57-95db-2fed4c7ddbec","Type":"ContainerDied","Data":"b58dc8193beaf3222ca8e249e8e65cb04ba60ac1e8c013cc5bde0651822b681c"} Nov 03 01:21:37.822697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.822672 5144 scope.go:122] "RemoveContainer" containerID="b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45" Nov 03 01:21:37.823153 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:37.823130 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45\": container with ID starting with b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45 not found: ID does not exist" containerID="b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45" Nov 03 01:21:37.823249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.823163 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45"} err="failed to get container status \"b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45\": rpc error: code = NotFound desc = could not find container \"b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45\": container with ID starting with b700ec6740ca6322957e63ab45a01048755b6da1b8b97c8bd4113f8a49f2ae45 not found: ID does not exist" Nov 03 01:21:37.858524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.858503 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 01:21:37.914070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.914047 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume\") pod \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " Nov 03 01:21:37.914220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.914078 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6\" (UniqueName: \"kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6\") pod \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\" (UID: \"ee41c01d-7380-4c57-95db-2fed4c7ddbec\") " Nov 03 01:21:37.914388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.914367 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume" pod "ee41c01d-7380-4c57-95db-2fed4c7ddbec" (UID: "ee41c01d-7380-4c57-95db-2fed4c7ddbec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:37.916316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.916293 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6" pod "ee41c01d-7380-4c57-95db-2fed4c7ddbec" (UID: "ee41c01d-7380-4c57-95db-2fed4c7ddbec"). InnerVolumeSpecName "kube-api-access-d78b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:37.972594 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:37.972567 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp"] Nov 03 01:21:38.015534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.015473 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee41c01d-7380-4c57-95db-2fed4c7ddbec-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.015534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.015499 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d78b6\" (UniqueName: \"kubernetes.io/projected/ee41c01d-7380-4c57-95db-2fed4c7ddbec-kube-api-access-d78b6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.022033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.022014 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b2e795-7c7d49d475-j8pzp"] Nov 03 01:21:38.706751 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.706726 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 01:21:38.820692 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.820668 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" Nov 03 01:21:38.820861 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.820660 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk" event={"ID":"ee41c01d-7380-4c57-95db-2fed4c7ddbec","Type":"ContainerDied","Data":"39f613fd9bd61aafe5b4e9378b6bdefae9b8c2eba2bc48f15ef6a042401d958f"} Nov 03 01:21:38.820861 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.820816 5144 scope.go:122] "RemoveContainer" containerID="b58dc8193beaf3222ca8e249e8e65cb04ba60ac1e8c013cc5bde0651822b681c" Nov 03 01:21:38.823519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823495 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.823675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823562 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.823675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823616 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.823675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823671 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.823852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823711 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.823852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823782 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.823852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823814 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.824014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823860 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.824014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.823933 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket\") pod \"5bf746e0-37b0-4325-b5b7-31465218f238\" (UID: \"5bf746e0-37b0-4325-b5b7-31465218f238\") " Nov 03 01:21:38.824618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.824590 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.824871 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.824849 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.824958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.824892 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.825314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.825080 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:38.825665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.825467 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.826147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.825832 5144 generic.go:356] "Generic (PLEG): container finished" podID="5bf746e0-37b0-4325-b5b7-31465218f238" containerID="c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8" exitCode=0 Nov 03 01:21:38.826147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.825876 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" event={"ID":"5bf746e0-37b0-4325-b5b7-31465218f238","Type":"ContainerDied","Data":"c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8"} Nov 03 01:21:38.826147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.825915 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" event={"ID":"5bf746e0-37b0-4325-b5b7-31465218f238","Type":"ContainerDied","Data":"e209917d58f4c47dbad1f2563541315e370bfcae1aad3a45be11731b0ff15fd9"} Nov 03 01:21:38.826147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.825989 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7" Nov 03 01:21:38.827823 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.827748 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:21:38.828940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.828909 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:21:38.830898 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.830866 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "kube-api-access-6gddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:38.838117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.833994 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token" pod "5bf746e0-37b0-4325-b5b7-31465218f238" (UID: "5bf746e0-37b0-4325-b5b7-31465218f238"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:38.849901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.849881 5144 scope.go:122] "RemoveContainer" containerID="c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8" Nov 03 01:21:38.861956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.861933 5144 scope.go:122] "RemoveContainer" containerID="0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22" Nov 03 01:21:38.871127 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.871111 5144 scope.go:122] "RemoveContainer" containerID="c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8" Nov 03 01:21:38.871365 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:38.871341 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8\": container with ID starting with c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8 not found: ID does not exist" containerID="c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8" Nov 03 01:21:38.871444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.871369 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8"} err="failed to get container status \"c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8\": rpc error: code = NotFound desc = could not find container \"c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8\": container with ID starting with c571b7ae0560bf335c9a064e3ae1fe8efdf2866732f6b07d2b7573fb3bb92dd8 not found: ID does not exist" Nov 03 01:21:38.871444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.871387 5144 scope.go:122] "RemoveContainer" containerID="0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22" Nov 03 01:21:38.871623 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:38.871609 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22\": container with ID starting with 0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22 not found: ID does not exist" containerID="0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22" Nov 03 01:21:38.871786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.871642 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22"} err="failed to get container status \"0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22\": rpc error: code = NotFound desc = could not find container \"0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22\": container with ID starting with 0d4beb45f03f286598283070186a3e40f3c212dba954cc8339a8b2c09dc25c22 not found: ID does not exist" Nov 03 01:21:38.926042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926004 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926046 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926062 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bf746e0-37b0-4325-b5b7-31465218f238-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926079 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926091 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926104 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926117 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gddb\" (UniqueName: \"kubernetes.io/projected/5bf746e0-37b0-4325-b5b7-31465218f238-kube-api-access-6gddb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926129 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bf746e0-37b0-4325-b5b7-31465218f238-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:38.926154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:38.926140 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bf746e0-37b0-4325-b5b7-31465218f238-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:39.038826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:39.038754 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk"] Nov 03 01:21:39.043752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:39.043721 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b2e795-76cd898d9f-mdzsk"] Nov 03 01:21:39.249069 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:39.249040 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7"] Nov 03 01:21:39.254458 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:39.254435 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-b2e795-openshift-default-778c45d857-8z6g7"] Nov 03 01:21:39.999529 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:39.999491 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" path="/var/lib/kubelet/pods/2a8e5d31-1e1c-4550-b520-33160ad876db/volumes" Nov 03 01:21:40.000061 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:39.999975 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" path="/var/lib/kubelet/pods/5bf746e0-37b0-4325-b5b7-31465218f238/volumes" Nov 03 01:21:40.000387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:40.000372 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" path="/var/lib/kubelet/pods/ee41c01d-7380-4c57-95db-2fed4c7ddbec/volumes" Nov 03 01:21:40.159116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:40.159076 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:40.963183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:40.963144 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd"] Nov 03 01:21:41.072115 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:41.072077 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq"] Nov 03 01:21:41.170866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:41.170832 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:42.159449 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:42.159412 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:21:45.964027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:45.963992 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerName="payload-processing" containerID="cri-o://d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf" gracePeriod=25 Nov 03 01:21:46.072534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:46.072504 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerName="payload-pre-processing" containerID="cri-o://82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333" gracePeriod=25 Nov 03 01:21:50.159244 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:50.159207 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:50.726963 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:50.726923 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:50.727159 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:50.727012 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:51.170384 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:51.170336 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:21:56.045440 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.045405 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp"] Nov 03 01:21:56.094969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.094945 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7effafee-vmptp"] Nov 03 01:21:56.773445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.773368 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 01:21:56.867682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.867659 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 01:21:56.907684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.907614 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume\") pod \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " Nov 03 01:21:56.907882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.907864 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q\" (UniqueName: \"kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q\") pod \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\" (UID: \"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6\") " Nov 03 01:21:56.907947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.907930 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume" pod "16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" (UID: "16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:56.908236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.908217 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:56.910075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.910053 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q" pod "16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" (UID: "16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6"). InnerVolumeSpecName "kube-api-access-g279q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:56.952869 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.952841 5144 generic.go:356] "Generic (PLEG): container finished" podID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerID="f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0" exitCode=0 Nov 03 01:21:56.952987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.952880 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" Nov 03 01:21:56.952987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.952918 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" event={"ID":"b5510541-f988-47ff-8d94-4a1bbe2144cf","Type":"ContainerDied","Data":"f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0"} Nov 03 01:21:56.952987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.952979 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw" event={"ID":"b5510541-f988-47ff-8d94-4a1bbe2144cf","Type":"ContainerDied","Data":"e83d959cb8d7baea504700766889be3e77016d69c7b17a63c4dedfbb56d687e8"} Nov 03 01:21:56.953176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.953012 5144 scope.go:122] "RemoveContainer" containerID="f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0" Nov 03 01:21:56.954441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.954419 5144 generic.go:356] "Generic (PLEG): container finished" podID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerID="05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0" exitCode=0 Nov 03 01:21:56.954569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.954459 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" Nov 03 01:21:56.954569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.954501 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" event={"ID":"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6","Type":"ContainerDied","Data":"05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0"} Nov 03 01:21:56.954569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.954540 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf" event={"ID":"16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6","Type":"ContainerDied","Data":"b7de6533ac3216867ff3460e0b684d722e9a945cd291a9e7d34b46d8d15dc9e0"} Nov 03 01:21:56.963808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.963789 5144 scope.go:122] "RemoveContainer" containerID="f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0" Nov 03 01:21:56.964060 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:56.964040 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0\": container with ID starting with f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0 not found: ID does not exist" containerID="f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0" Nov 03 01:21:56.964120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.964067 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0"} err="failed to get container status \"f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0\": rpc error: code = NotFound desc = could not find container \"f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0\": container with ID starting with f48147ca3b9071df73802db112be5f17eba961555f2479899d5388589aa8c9c0 not found: ID does not exist" Nov 03 01:21:56.964120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.964082 5144 scope.go:122] "RemoveContainer" containerID="05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0" Nov 03 01:21:56.972755 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.972736 5144 scope.go:122] "RemoveContainer" containerID="05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0" Nov 03 01:21:56.973014 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:21:56.972997 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0\": container with ID starting with 05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0 not found: ID does not exist" containerID="05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0" Nov 03 01:21:56.973090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:56.973020 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0"} err="failed to get container status \"05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0\": rpc error: code = NotFound desc = could not find container \"05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0\": container with ID starting with 05e93056fff05107a3d6d8d538dad6e5e1ceae37f3c732a8e861c57ec3ec0ff0 not found: ID does not exist" Nov 03 01:21:57.008915 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.008890 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume\") pod \"b5510541-f988-47ff-8d94-4a1bbe2144cf\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " Nov 03 01:21:57.009055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.008963 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5\" (UniqueName: \"kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5\") pod \"b5510541-f988-47ff-8d94-4a1bbe2144cf\" (UID: \"b5510541-f988-47ff-8d94-4a1bbe2144cf\") " Nov 03 01:21:57.009273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.009255 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g279q\" (UniqueName: \"kubernetes.io/projected/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6-kube-api-access-g279q\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:57.009345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.009264 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume" pod "b5510541-f988-47ff-8d94-4a1bbe2144cf" (UID: "b5510541-f988-47ff-8d94-4a1bbe2144cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:21:57.011438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.011410 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5" pod "b5510541-f988-47ff-8d94-4a1bbe2144cf" (UID: "b5510541-f988-47ff-8d94-4a1bbe2144cf"). InnerVolumeSpecName "kube-api-access-pngh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:21:57.110136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.110040 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pngh5\" (UniqueName: \"kubernetes.io/projected/b5510541-f988-47ff-8d94-4a1bbe2144cf-kube-api-access-pngh5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:57.110136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.110068 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5510541-f988-47ff-8d94-4a1bbe2144cf-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:21:57.110874 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.110851 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf"] Nov 03 01:21:57.116674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.116652 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-048135-69b7885d9c-q5pgf"] Nov 03 01:21:57.373272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.373154 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw"] Nov 03 01:21:57.376991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.376965 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-048135-77486574b7-jd8mw"] Nov 03 01:21:57.995216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.995188 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" path="/var/lib/kubelet/pods/16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6/volumes" Nov 03 01:21:57.995532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.995519 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" path="/var/lib/kubelet/pods/b5510541-f988-47ff-8d94-4a1bbe2144cf/volumes" Nov 03 01:21:57.995881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:21:57.995868 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba88532-0bc8-4f2b-bb18-63e016f1aeb8" path="/var/lib/kubelet/pods/bba88532-0bc8-4f2b-bb18-63e016f1aeb8/volumes" Nov 03 01:22:00.727278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:00.727238 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:22:00.727736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:00.727287 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:22:02.192268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:02.192235 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5"] Nov 03 01:22:02.192757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:02.192433 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" containerID="cri-o://07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe" gracePeriod=30 Nov 03 01:22:03.332269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332227 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg"] Nov 03 01:22:03.332759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332743 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" Nov 03 01:22:03.332815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332760 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" Nov 03 01:22:03.332815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332773 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" Nov 03 01:22:03.332815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332778 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" Nov 03 01:22:03.332815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332790 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerName="payload-pre-processing" Nov 03 01:22:03.332815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332796 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerName="payload-pre-processing" Nov 03 01:22:03.332815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332814 5144 cpu_manager.go:414] "containerMap: removing container" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerName="payload-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332819 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerName="payload-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332828 5144 cpu_manager.go:414] "containerMap: removing container" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332832 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332845 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerName="payload-pre-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332851 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerName="payload-pre-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332860 5144 cpu_manager.go:414] "containerMap: removing container" podUID="ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" containerName="revoke-keys" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332864 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" containerName="revoke-keys" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332876 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerName="payload-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332880 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerName="payload-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332943 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332954 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee41c01d-7380-4c57-95db-2fed4c7ddbec" containerName="payload-pre-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332962 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="16fbaa05-5dbc-4b8f-a70a-41cbef2a13d6" containerName="payload-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332967 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="32861b2c-92a4-4763-83eb-9b8e0eb3e083" containerName="main" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332974 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5510541-f988-47ff-8d94-4a1bbe2144cf" containerName="payload-pre-processing" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332979 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" containerName="revoke-keys" Nov 03 01:22:03.333004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.332985 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8e5d31-1e1c-4550-b520-33160ad876db" containerName="payload-processing" Nov 03 01:22:03.337854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.337835 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.392248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.392215 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg"] Nov 03 01:22:03.437760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.437720 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:22:03.438210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.437808 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:22:03.472972 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.472940 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.473124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.473004 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmn7w\" (UniqueName: \"kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.473124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.473091 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.574412 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.574351 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.574570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.574459 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmn7w\" (UniqueName: \"kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.574570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.574487 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.575247 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.575218 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.576933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.576907 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.639273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.639198 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmn7w\" (UniqueName: \"kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w\") pod \"maas-api-revoke-sub-35e71f1d-v5ldg\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:03.652069 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:03.652036 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:04.027375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:04.027349 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg"] Nov 03 01:22:04.029141 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:22:04.029116 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581e7c72_05ed_4081_ba5c_76e46ae82e08.slice/crio-532b3696dfe4f45facd644a8bde107fcdc609f7861e48dae2fcfef7bdfd1830b WatchSource:0}: Error finding container 532b3696dfe4f45facd644a8bde107fcdc609f7861e48dae2fcfef7bdfd1830b: Status 404 returned error can't find the container with id 532b3696dfe4f45facd644a8bde107fcdc609f7861e48dae2fcfef7bdfd1830b Nov 03 01:22:05.010863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:05.010820 5144 generic.go:356] "Generic (PLEG): container finished" podID="581e7c72-05ed-4081-ba5c-76e46ae82e08" containerID="1f5b3dd1488777a0af3cd548e1cd6dfdab3aa9c8e93577d221a103cd614c0e27" exitCode=0 Nov 03 01:22:05.011330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:05.010903 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" event={"ID":"581e7c72-05ed-4081-ba5c-76e46ae82e08","Type":"ContainerDied","Data":"1f5b3dd1488777a0af3cd548e1cd6dfdab3aa9c8e93577d221a103cd614c0e27"} Nov 03 01:22:05.011330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:05.010977 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" event={"ID":"581e7c72-05ed-4081-ba5c-76e46ae82e08","Type":"ContainerStarted","Data":"532b3696dfe4f45facd644a8bde107fcdc609f7861e48dae2fcfef7bdfd1830b"} Nov 03 01:22:06.051741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.051706 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp"] Nov 03 01:22:06.100035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.100007 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bafb469-bdkwp"] Nov 03 01:22:06.205030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.205007 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:06.237452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.237422 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 01:22:06.302096 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.302066 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w\" (UniqueName: \"kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w\") pod \"581e7c72-05ed-4081-ba5c-76e46ae82e08\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " Nov 03 01:22:06.302289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.302194 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca\") pod \"581e7c72-05ed-4081-ba5c-76e46ae82e08\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " Nov 03 01:22:06.302289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.302256 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca\") pod \"581e7c72-05ed-4081-ba5c-76e46ae82e08\" (UID: \"581e7c72-05ed-4081-ba5c-76e46ae82e08\") " Nov 03 01:22:06.302650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.302603 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca" pod "581e7c72-05ed-4081-ba5c-76e46ae82e08" (UID: "581e7c72-05ed-4081-ba5c-76e46ae82e08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:06.305116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.305091 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca" pod "581e7c72-05ed-4081-ba5c-76e46ae82e08" (UID: "581e7c72-05ed-4081-ba5c-76e46ae82e08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:06.305232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.305133 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w" pod "581e7c72-05ed-4081-ba5c-76e46ae82e08" (UID: "581e7c72-05ed-4081-ba5c-76e46ae82e08"). InnerVolumeSpecName "kube-api-access-fmn7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:06.337387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.337359 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 01:22:06.403251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.403218 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume\") pod \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " Nov 03 01:22:06.403458 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.403357 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89\" (UniqueName: \"kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89\") pod \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\" (UID: \"b25748e5-b0fa-47a4-948a-170d3ef2f6e9\") " Nov 03 01:22:06.403746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.403724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume" pod "b25748e5-b0fa-47a4-948a-170d3ef2f6e9" (UID: "b25748e5-b0fa-47a4-948a-170d3ef2f6e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:06.403806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.403737 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmn7w\" (UniqueName: \"kubernetes.io/projected/581e7c72-05ed-4081-ba5c-76e46ae82e08-kube-api-access-fmn7w\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:06.403806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.403772 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:06.403806 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.403784 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581e7c72-05ed-4081-ba5c-76e46ae82e08-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:06.405697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.405672 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89" pod "b25748e5-b0fa-47a4-948a-170d3ef2f6e9" (UID: "b25748e5-b0fa-47a4-948a-170d3ef2f6e9"). InnerVolumeSpecName "kube-api-access-94r89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:06.504580 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.504550 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume\") pod \"7d4c4328-fc5b-4eba-afad-ac674b898816\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " Nov 03 01:22:06.504787 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.504678 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh\" (UniqueName: \"kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh\") pod \"7d4c4328-fc5b-4eba-afad-ac674b898816\" (UID: \"7d4c4328-fc5b-4eba-afad-ac674b898816\") " Nov 03 01:22:06.504941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.504928 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:06.505076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.504941 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94r89\" (UniqueName: \"kubernetes.io/projected/b25748e5-b0fa-47a4-948a-170d3ef2f6e9-kube-api-access-94r89\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:06.505076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.504988 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume" pod "7d4c4328-fc5b-4eba-afad-ac674b898816" (UID: "7d4c4328-fc5b-4eba-afad-ac674b898816"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:06.506880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.506858 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh" pod "7d4c4328-fc5b-4eba-afad-ac674b898816" (UID: "7d4c4328-fc5b-4eba-afad-ac674b898816"). InnerVolumeSpecName "kube-api-access-ddrmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:06.605891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.605851 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d4c4328-fc5b-4eba-afad-ac674b898816-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:06.605891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:06.605894 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddrmh\" (UniqueName: \"kubernetes.io/projected/7d4c4328-fc5b-4eba-afad-ac674b898816-kube-api-access-ddrmh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.025847 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.025812 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" Nov 03 01:22:07.025847 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.025843 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg" event={"ID":"581e7c72-05ed-4081-ba5c-76e46ae82e08","Type":"ContainerDied","Data":"532b3696dfe4f45facd644a8bde107fcdc609f7861e48dae2fcfef7bdfd1830b"} Nov 03 01:22:07.026126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.025911 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="532b3696dfe4f45facd644a8bde107fcdc609f7861e48dae2fcfef7bdfd1830b" Nov 03 01:22:07.027358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.027325 5144 generic.go:356] "Generic (PLEG): container finished" podID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerID="d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf" exitCode=0 Nov 03 01:22:07.027358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.027359 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" Nov 03 01:22:07.027583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.027413 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" event={"ID":"b25748e5-b0fa-47a4-948a-170d3ef2f6e9","Type":"ContainerDied","Data":"d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf"} Nov 03 01:22:07.027583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.027472 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd" event={"ID":"b25748e5-b0fa-47a4-948a-170d3ef2f6e9","Type":"ContainerDied","Data":"29b0472d198724f01ecf632ec4977b1c21766e1120aa2ae2cafb1b8374fe84fd"} Nov 03 01:22:07.027583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.027498 5144 scope.go:122] "RemoveContainer" containerID="d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf" Nov 03 01:22:07.028973 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.028932 5144 generic.go:356] "Generic (PLEG): container finished" podID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerID="82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333" exitCode=0 Nov 03 01:22:07.029045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.028976 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" Nov 03 01:22:07.029045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.028994 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" event={"ID":"7d4c4328-fc5b-4eba-afad-ac674b898816","Type":"ContainerDied","Data":"82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333"} Nov 03 01:22:07.029045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.029026 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq" event={"ID":"7d4c4328-fc5b-4eba-afad-ac674b898816","Type":"ContainerDied","Data":"0a7975b674f3ae3ab53e0ee96763408d8d2901b6a21c735300485ad4dc8befab"} Nov 03 01:22:07.040038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.040015 5144 scope.go:122] "RemoveContainer" containerID="d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf" Nov 03 01:22:07.040328 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:07.040310 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf\": container with ID starting with d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf not found: ID does not exist" containerID="d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf" Nov 03 01:22:07.040386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.040337 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf"} err="failed to get container status \"d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf\": rpc error: code = NotFound desc = could not find container \"d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf\": container with ID starting with d89d2a4349ab387c1f127a43d361336a056e9f045b38b32b8f5cd3b7ce6f6bcf not found: ID does not exist" Nov 03 01:22:07.040386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.040353 5144 scope.go:122] "RemoveContainer" containerID="82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333" Nov 03 01:22:07.050378 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.050350 5144 scope.go:122] "RemoveContainer" containerID="82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333" Nov 03 01:22:07.050684 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:07.050664 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333\": container with ID starting with 82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333 not found: ID does not exist" containerID="82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333" Nov 03 01:22:07.050757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.050696 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333"} err="failed to get container status \"82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333\": rpc error: code = NotFound desc = could not find container \"82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333\": container with ID starting with 82abc26ddceb158b83f2573eae2b7eb14dba2005b2851842854719b0e46b2333 not found: ID does not exist" Nov 03 01:22:07.191862 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.191826 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd"] Nov 03 01:22:07.241673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.241601 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-904a34-9d8c9d4b7-cwlnd"] Nov 03 01:22:07.304280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.304247 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq"] Nov 03 01:22:07.307000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.306974 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-904a34-5fcb4d5d7-fv6lq"] Nov 03 01:22:07.450118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.450093 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 01:22:07.614702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614667 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.614894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614727 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.614894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614756 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.614894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614773 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.614894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614822 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.614894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614878 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.615192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614947 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.615192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.614982 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.615192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.615043 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs\") pod \"6052680c-30b2-44f4-ac41-f24f33dc8fca\" (UID: \"6052680c-30b2-44f4-ac41-f24f33dc8fca\") " Nov 03 01:22:07.615360 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.615320 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:07.615360 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.615336 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:07.615479 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.615432 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:07.615587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.615566 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:07.615889 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.615862 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:07.617375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.617346 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:07.617519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.617496 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:07.617519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.617509 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "kube-api-access-z8ddg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:07.617610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.617581 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token" pod "6052680c-30b2-44f4-ac41-f24f33dc8fca" (UID: "6052680c-30b2-44f4-ac41-f24f33dc8fca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:07.716559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716525 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716552 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6052680c-30b2-44f4-ac41-f24f33dc8fca-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716563 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716573 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716583 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716591 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8ddg\" (UniqueName: \"kubernetes.io/projected/6052680c-30b2-44f4-ac41-f24f33dc8fca-kube-api-access-z8ddg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716602 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716611 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6052680c-30b2-44f4-ac41-f24f33dc8fca-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.716851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.716619 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6052680c-30b2-44f4-ac41-f24f33dc8fca-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:07.994887 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.994800 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" path="/var/lib/kubelet/pods/7d4c4328-fc5b-4eba-afad-ac674b898816/volumes" Nov 03 01:22:07.995243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.995227 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cd4b33e-cf3b-4323-a101-085bfc19239f" path="/var/lib/kubelet/pods/8cd4b33e-cf3b-4323-a101-085bfc19239f/volumes" Nov 03 01:22:07.995583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:07.995569 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" path="/var/lib/kubelet/pods/b25748e5-b0fa-47a4-948a-170d3ef2f6e9/volumes" Nov 03 01:22:08.038754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.038718 5144 generic.go:356] "Generic (PLEG): container finished" podID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerID="07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe" exitCode=0 Nov 03 01:22:08.038958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.038778 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" Nov 03 01:22:08.038958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.038802 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" event={"ID":"6052680c-30b2-44f4-ac41-f24f33dc8fca","Type":"ContainerDied","Data":"07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe"} Nov 03 01:22:08.038958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.038864 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5" event={"ID":"6052680c-30b2-44f4-ac41-f24f33dc8fca","Type":"ContainerDied","Data":"545e307866bd9b7214ec3a22d129e4705d530068ea2531906eccfbc30f3e55be"} Nov 03 01:22:08.038958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.038888 5144 scope.go:122] "RemoveContainer" containerID="07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe" Nov 03 01:22:08.049258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.049237 5144 scope.go:122] "RemoveContainer" containerID="a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03" Nov 03 01:22:08.058869 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.058847 5144 scope.go:122] "RemoveContainer" containerID="07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe" Nov 03 01:22:08.059144 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:08.059121 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe\": container with ID starting with 07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe not found: ID does not exist" containerID="07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe" Nov 03 01:22:08.059203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.059153 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe"} err="failed to get container status \"07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe\": rpc error: code = NotFound desc = could not find container \"07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe\": container with ID starting with 07edab438c381526a585e31146e0ed7fb0fe6aea148216b611f306a89ef74ffe not found: ID does not exist" Nov 03 01:22:08.059203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.059171 5144 scope.go:122] "RemoveContainer" containerID="a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03" Nov 03 01:22:08.059408 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:08.059394 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03\": container with ID starting with a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03 not found: ID does not exist" containerID="a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03" Nov 03 01:22:08.059451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.059413 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03"} err="failed to get container status \"a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03\": rpc error: code = NotFound desc = could not find container \"a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03\": container with ID starting with a3fb5e3e3d8b5ad580dd97915871d11f9556655402910a61ea4d9f2f81453d03 not found: ID does not exist" Nov 03 01:22:08.202981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.202933 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5"] Nov 03 01:22:08.252304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:08.252212 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-904a34-openshift-default-9967459c-gd7l5"] Nov 03 01:22:09.997046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:09.997013 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" path="/var/lib/kubelet/pods/6052680c-30b2-44f4-ac41-f24f33dc8fca/volumes" Nov 03 01:22:12.159406 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:12.159368 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:22:14.119325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:14.119292 5144 scope.go:122] "RemoveContainer" containerID="976b0e537b59f74507c9c42d2cbecba9c7d314e9a563ed3f96fb771a6ac84301" Nov 03 01:22:14.131332 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:14.131309 5144 scope.go:122] "RemoveContainer" containerID="d35c0b689aa208f90c1d0f3e415973c55c50d6c155d9d3b52b5b534a4e765aee" Nov 03 01:22:18.044958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:18.044915 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954"] Nov 03 01:22:18.094728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:18.094692 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-793e935c-pz954"] Nov 03 01:22:19.995484 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:19.995403 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="021afe20-e654-48e1-8548-0336da129859" path="/var/lib/kubelet/pods/021afe20-e654-48e1-8548-0336da129859/volumes" Nov 03 01:22:20.100151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100113 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt"] Nov 03 01:22:20.100564 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100552 5144 cpu_manager.go:414] "containerMap: removing container" podUID="581e7c72-05ed-4081-ba5c-76e46ae82e08" containerName="revoke-keys" Nov 03 01:22:20.100618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100565 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581e7c72-05ed-4081-ba5c-76e46ae82e08" containerName="revoke-keys" Nov 03 01:22:20.100618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100601 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" Nov 03 01:22:20.100618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100606 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" Nov 03 01:22:20.100618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100614 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerName="payload-processing" Nov 03 01:22:20.100618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100619 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerName="payload-processing" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100646 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerName="payload-pre-processing" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100651 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerName="payload-pre-processing" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100659 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100664 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100728 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100735 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b25748e5-b0fa-47a4-948a-170d3ef2f6e9" containerName="payload-processing" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100742 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d4c4328-fc5b-4eba-afad-ac674b898816" containerName="payload-pre-processing" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100749 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6052680c-30b2-44f4-ac41-f24f33dc8fca" containerName="istio-proxy" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100756 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="581e7c72-05ed-4081-ba5c-76e46ae82e08" containerName="revoke-keys" Nov 03 01:22:20.100805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.100763 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf746e0-37b0-4325-b5b7-31465218f238" containerName="istio-proxy" Nov 03 01:22:20.103637 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.103610 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.130005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.129975 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.130177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.130107 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.130177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.130158 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hs9\" (UniqueName: \"kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.160148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.160120 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt"] Nov 03 01:22:20.204948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.204910 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:22:20.204948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.204935 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9mt64\"" Nov 03 01:22:20.230870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.230833 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.230870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.230876 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hs9\" (UniqueName: \"kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.231113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.230930 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.231643 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.231572 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.233389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.233363 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.287197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.287165 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hs9\" (UniqueName: \"kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9\") pod \"maas-api-revoke-sub-ea3c67db-drmrt\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.425607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.425571 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:20.800535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:20.800509 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt"] Nov 03 01:22:20.801850 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:22:20.801819 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode759c769_df16_4098_9705_74507aaddf9b.slice/crio-0f061c1c5c42bd2f03d2727bf910a6c9c0b6df5a35225d4a836d914713c74271 WatchSource:0}: Error finding container 0f061c1c5c42bd2f03d2727bf910a6c9c0b6df5a35225d4a836d914713c74271: Status 404 returned error can't find the container with id 0f061c1c5c42bd2f03d2727bf910a6c9c0b6df5a35225d4a836d914713c74271 Nov 03 01:22:21.130870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:21.130842 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" event={"ID":"e759c769-df16-4098-9705-74507aaddf9b","Type":"ContainerStarted","Data":"0f061c1c5c42bd2f03d2727bf910a6c9c0b6df5a35225d4a836d914713c74271"} Nov 03 01:22:21.187949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:21.187911 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm"] Nov 03 01:22:21.295248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:21.295204 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2"] Nov 03 01:22:22.139200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:22.139167 5144 generic.go:356] "Generic (PLEG): container finished" podID="e759c769-df16-4098-9705-74507aaddf9b" containerID="a56cd0c7d684b6ed1e02b3e374adadbc2196d2d9d8a167876f8e3f329c726d72" exitCode=0 Nov 03 01:22:22.139691 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:22.139250 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" event={"ID":"e759c769-df16-4098-9705-74507aaddf9b","Type":"ContainerDied","Data":"a56cd0c7d684b6ed1e02b3e374adadbc2196d2d9d8a167876f8e3f329c726d72"} Nov 03 01:22:23.297239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.297205 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:23.361029 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.360997 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca\") pod \"e759c769-df16-4098-9705-74507aaddf9b\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " Nov 03 01:22:23.361254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.361110 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9\" (UniqueName: \"kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9\") pod \"e759c769-df16-4098-9705-74507aaddf9b\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " Nov 03 01:22:23.361254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.361149 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca\") pod \"e759c769-df16-4098-9705-74507aaddf9b\" (UID: \"e759c769-df16-4098-9705-74507aaddf9b\") " Nov 03 01:22:23.361593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.361562 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca" pod "e759c769-df16-4098-9705-74507aaddf9b" (UID: "e759c769-df16-4098-9705-74507aaddf9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:23.363417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.363377 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca" pod "e759c769-df16-4098-9705-74507aaddf9b" (UID: "e759c769-df16-4098-9705-74507aaddf9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:22:23.363676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.363608 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9" pod "e759c769-df16-4098-9705-74507aaddf9b" (UID: "e759c769-df16-4098-9705-74507aaddf9b"). InnerVolumeSpecName "kube-api-access-86hs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:23.462656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.462535 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e759c769-df16-4098-9705-74507aaddf9b-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.462656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.462568 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hs9\" (UniqueName: \"kubernetes.io/projected/e759c769-df16-4098-9705-74507aaddf9b-kube-api-access-86hs9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:23.462656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:23.462580 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e759c769-df16-4098-9705-74507aaddf9b-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:24.155826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:24.155789 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" Nov 03 01:22:24.156004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:24.155829 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt" event={"ID":"e759c769-df16-4098-9705-74507aaddf9b","Type":"ContainerDied","Data":"0f061c1c5c42bd2f03d2727bf910a6c9c0b6df5a35225d4a836d914713c74271"} Nov 03 01:22:24.156004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:24.155874 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f061c1c5c42bd2f03d2727bf910a6c9c0b6df5a35225d4a836d914713c74271" Nov 03 01:22:26.188913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:26.188876 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerName="payload-processing" containerID="cri-o://c133cbc995a9bd4d968360e788b3904efc16fa5dc34f3067dbce02245d5beb5a" gracePeriod=25 Nov 03 01:22:26.296489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:26.296441 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerName="payload-pre-processing" containerID="cri-o://c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99" gracePeriod=25 Nov 03 01:22:27.341823 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:27.341772 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:22:30.054473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:30.054432 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h"] Nov 03 01:22:30.104374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:30.104342 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e72c737d-rjs6h"] Nov 03 01:22:31.995557 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:31.995509 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7de899b5-3ced-4c50-9a52-595ef4764915" path="/var/lib/kubelet/pods/7de899b5-3ced-4c50-9a52-595ef4764915/volumes" Nov 03 01:22:35.272075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:35.272037 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:22:37.293650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:37.293604 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:22:42.161486 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:42.161448 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:22:45.272078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:45.272043 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:22:46.310859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.310822 5144 generic.go:356] "Generic (PLEG): container finished" podID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerID="c133cbc995a9bd4d968360e788b3904efc16fa5dc34f3067dbce02245d5beb5a" exitCode=0 Nov 03 01:22:46.311293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.310891 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" event={"ID":"803d1d10-ac58-4c88-98ad-27c52fe1de02","Type":"ContainerDied","Data":"c133cbc995a9bd4d968360e788b3904efc16fa5dc34f3067dbce02245d5beb5a"} Nov 03 01:22:46.460414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.460338 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 01:22:46.555510 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.555483 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 01:22:46.621425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.621391 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh\" (UniqueName: \"kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh\") pod \"803d1d10-ac58-4c88-98ad-27c52fe1de02\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " Nov 03 01:22:46.621613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.621499 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume\") pod \"803d1d10-ac58-4c88-98ad-27c52fe1de02\" (UID: \"803d1d10-ac58-4c88-98ad-27c52fe1de02\") " Nov 03 01:22:46.621948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.621907 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume" pod "803d1d10-ac58-4c88-98ad-27c52fe1de02" (UID: "803d1d10-ac58-4c88-98ad-27c52fe1de02"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:46.623869 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.623846 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh" pod "803d1d10-ac58-4c88-98ad-27c52fe1de02" (UID: "803d1d10-ac58-4c88-98ad-27c52fe1de02"). InnerVolumeSpecName "kube-api-access-tlpnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:46.722722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.722611 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume\") pod \"521293d7-4cb2-4ad3-b258-803887e4fbdf\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " Nov 03 01:22:46.722722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.722717 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w\" (UniqueName: \"kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w\") pod \"521293d7-4cb2-4ad3-b258-803887e4fbdf\" (UID: \"521293d7-4cb2-4ad3-b258-803887e4fbdf\") " Nov 03 01:22:46.723080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.723054 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume" pod "521293d7-4cb2-4ad3-b258-803887e4fbdf" (UID: "521293d7-4cb2-4ad3-b258-803887e4fbdf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:46.723148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.723126 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/803d1d10-ac58-4c88-98ad-27c52fe1de02-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:46.723194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.723148 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlpnh\" (UniqueName: \"kubernetes.io/projected/803d1d10-ac58-4c88-98ad-27c52fe1de02-kube-api-access-tlpnh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:46.725029 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.725004 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w" pod "521293d7-4cb2-4ad3-b258-803887e4fbdf" (UID: "521293d7-4cb2-4ad3-b258-803887e4fbdf"). InnerVolumeSpecName "kube-api-access-2qc8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:46.824048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.824013 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/521293d7-4cb2-4ad3-b258-803887e4fbdf-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:46.824048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:46.824045 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qc8w\" (UniqueName: \"kubernetes.io/projected/521293d7-4cb2-4ad3-b258-803887e4fbdf-kube-api-access-2qc8w\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:47.318410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.318346 5144 generic.go:356] "Generic (PLEG): container finished" podID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerID="c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99" exitCode=0 Nov 03 01:22:47.318901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.318418 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" Nov 03 01:22:47.318901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.318447 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" event={"ID":"521293d7-4cb2-4ad3-b258-803887e4fbdf","Type":"ContainerDied","Data":"c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99"} Nov 03 01:22:47.318901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.318507 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2" event={"ID":"521293d7-4cb2-4ad3-b258-803887e4fbdf","Type":"ContainerDied","Data":"634872c5ae4906d9ce1fffe3a39515a876051dd83af8f16d6c18dfa27f95099c"} Nov 03 01:22:47.318901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.318537 5144 scope.go:122] "RemoveContainer" containerID="c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99" Nov 03 01:22:47.320081 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.320066 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" Nov 03 01:22:47.320174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.320109 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm" event={"ID":"803d1d10-ac58-4c88-98ad-27c52fe1de02","Type":"ContainerDied","Data":"847d97f312269c9fb0f6f76f8e6ca317722d4f8940d6db22c18bec87277f2d5a"} Nov 03 01:22:47.329867 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.329603 5144 scope.go:122] "RemoveContainer" containerID="c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99" Nov 03 01:22:47.330010 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:47.329984 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99\": container with ID starting with c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99 not found: ID does not exist" containerID="c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99" Nov 03 01:22:47.330070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.330027 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99"} err="failed to get container status \"c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99\": rpc error: code = NotFound desc = could not find container \"c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99\": container with ID starting with c8b07ebbde5719acd3db3cef0a07b71780c6ac127ab7c4d05394eef37390da99 not found: ID does not exist" Nov 03 01:22:47.330070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.330049 5144 scope.go:122] "RemoveContainer" containerID="c133cbc995a9bd4d968360e788b3904efc16fa5dc34f3067dbce02245d5beb5a" Nov 03 01:22:47.482015 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.481924 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm"] Nov 03 01:22:47.487243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.487218 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-e28e1a-5ff9554777-qr9rm"] Nov 03 01:22:47.596157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.596113 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2"] Nov 03 01:22:47.602209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.602175 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-e28e1a-84fd756dc5-fjvn2"] Nov 03 01:22:47.995565 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.995529 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" path="/var/lib/kubelet/pods/521293d7-4cb2-4ad3-b258-803887e4fbdf/volumes" Nov 03 01:22:47.996229 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:47.996202 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" path="/var/lib/kubelet/pods/803d1d10-ac58-4c88-98ad-27c52fe1de02/volumes" Nov 03 01:22:48.590237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:48.590205 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8"] Nov 03 01:22:48.590701 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:48.590460 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" containerID="cri-o://90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1" gracePeriod=30 Nov 03 01:22:49.584388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:49.584351 5144 patch_prober.go:29] interesting pod/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:22:49.584706 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:49.584426 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:22:49.809230 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:49.809187 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64cfdbc956-4ltf2_a2768863-e2e0-46bd-9369-58c6a8a140d6/authorino/0.log" Nov 03 01:22:52.539747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:52.539713 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-wkpw5_4eb2b0f0-fb57-4948-b3a2-4d9094b4992c/manager/0.log" Nov 03 01:22:53.870441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.870348 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 01:22:53.986707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.986673 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.986937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.986721 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.986937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.986790 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.986937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.986860 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.986937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.986904 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.986937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.986930 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.987216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987009 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.987216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987046 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.987216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987072 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:53.987216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987086 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy\") pod \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\" (UID: \"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363\") " Nov 03 01:22:53.987216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987120 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:53.987470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987370 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:53.987470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987388 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:53.987932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987876 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:53.988073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.987954 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:53.988452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.988404 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:22:53.990209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.989771 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "kube-api-access-8ws9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:53.990489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.990461 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:22:53.990759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.990724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:22:53.991300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:53.991263 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy" pod "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" (UID: "8e28b6f2-d0b2-4346-9a0b-e706d0ac3363"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:22:54.087998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.087966 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.087998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.087994 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.087998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.088003 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.087998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.088011 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.088347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.088021 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.088347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.088030 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.088347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.088039 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ws9x\" (UniqueName: \"kubernetes.io/projected/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363-kube-api-access-8ws9x\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:22:54.165599 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.165520 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67cc775cd9-n49hj_af51a4fd-3ae1-46ac-b602-208c8a54c058/manager/0.log" Nov 03 01:22:54.370128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.370095 5144 generic.go:356] "Generic (PLEG): container finished" podID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerID="90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1" exitCode=0 Nov 03 01:22:54.370337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.370155 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" event={"ID":"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363","Type":"ContainerDied","Data":"90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1"} Nov 03 01:22:54.370337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.370178 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" Nov 03 01:22:54.370337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.370198 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8" event={"ID":"8e28b6f2-d0b2-4346-9a0b-e706d0ac3363","Type":"ContainerDied","Data":"6fd31e87a8468d5bd96eddea59e543549a0e6db27617147bdfd97eabf79774db"} Nov 03 01:22:54.370337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.370219 5144 scope.go:122] "RemoveContainer" containerID="90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1" Nov 03 01:22:54.380214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.380191 5144 scope.go:122] "RemoveContainer" containerID="5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e" Nov 03 01:22:54.389217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.389191 5144 scope.go:122] "RemoveContainer" containerID="90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1" Nov 03 01:22:54.389500 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:54.389477 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1\": container with ID starting with 90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1 not found: ID does not exist" containerID="90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1" Nov 03 01:22:54.389559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.389511 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1"} err="failed to get container status \"90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1\": rpc error: code = NotFound desc = could not find container \"90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1\": container with ID starting with 90ab37a70b08bf19fd53e630bbaeda59d0c15dfc4ee89926efbe5aee39f359d1 not found: ID does not exist" Nov 03 01:22:54.389559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.389536 5144 scope.go:122] "RemoveContainer" containerID="5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e" Nov 03 01:22:54.389792 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:22:54.389772 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e\": container with ID starting with 5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e not found: ID does not exist" containerID="5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e" Nov 03 01:22:54.389866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.389804 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e"} err="failed to get container status \"5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e\": rpc error: code = NotFound desc = could not find container \"5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e\": container with ID starting with 5983457cfbd8256994ebaf0b5e082702d47372004a1e3e2de2d9b93765c0e02e not found: ID does not exist" Nov 03 01:22:54.534933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.534901 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8"] Nov 03 01:22:54.537980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:54.537957 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-e28e1a-openshift-default-8664d896d7-9r9l8"] Nov 03 01:22:55.808968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:55.808942 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8q225_2dc160d5-869c-4f83-9abc-0f3a0018740f/server/0.log" Nov 03 01:22:55.995352 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:55.995318 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" path="/var/lib/kubelet/pods/8e28b6f2-d0b2-4346-9a0b-e706d0ac3363/volumes" Nov 03 01:22:56.344233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:22:56.344207 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-m2z5c_916b609e-7a1f-4137-83b5-424605bc72b0/proxy/0.log" Nov 03 01:23:01.322089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322005 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg"] Nov 03 01:23:01.322528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322484 5144 cpu_manager.go:414] "containerMap: removing container" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerName="payload-pre-processing" Nov 03 01:23:01.322528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322497 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerName="payload-pre-processing" Nov 03 01:23:01.322528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322509 5144 cpu_manager.go:414] "containerMap: removing container" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerName="payload-processing" Nov 03 01:23:01.322528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322514 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerName="payload-processing" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322533 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322539 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322547 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e759c769-df16-4098-9705-74507aaddf9b" containerName="revoke-keys" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322553 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e759c769-df16-4098-9705-74507aaddf9b" containerName="revoke-keys" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322566 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322573 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322719 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e759c769-df16-4098-9705-74507aaddf9b" containerName="revoke-keys" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322735 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322753 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="521293d7-4cb2-4ad3-b258-803887e4fbdf" containerName="payload-pre-processing" Nov 03 01:23:01.322808 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.322764 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="803d1d10-ac58-4c88-98ad-27c52fe1de02" containerName="payload-processing" Nov 03 01:23:01.327532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.327516 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345166 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345216 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgcs\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345238 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345413 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345354 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345413 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345372 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345531 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345457 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345531 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345495 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345533 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.345657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.345578 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.406668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.406618 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg"] Nov 03 01:23:01.432113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.432087 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-337498b9-openshift-default-dockercfg-w6sb9\"" Nov 03 01:23:01.446067 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446031 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446074 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446105 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446145 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446174 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446206 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446280 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446351 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgcs\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446388 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446473 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446543 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.446892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.446793 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.447582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.447049 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.447582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.447157 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.449502 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.449478 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.449964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.449945 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.502576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.502551 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.503295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.503266 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgcs\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs\") pod \"e2e-ait-337498b9-openshift-default-645db8f455-t2lcg\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:01.640710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:01.640608 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:02.069924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.069896 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg"] Nov 03 01:23:02.070340 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:23:02.070316 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eee358e_77c5_4fb9_b377_ecac27b13e0f.slice/crio-73c0e6bf1cbf6b6718b61e8e3f3be7a03e2ce22ce88705321761dcd8730ae655 WatchSource:0}: Error finding container 73c0e6bf1cbf6b6718b61e8e3f3be7a03e2ce22ce88705321761dcd8730ae655: Status 404 returned error can't find the container with id 73c0e6bf1cbf6b6718b61e8e3f3be7a03e2ce22ce88705321761dcd8730ae655 Nov 03 01:23:02.072299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.072269 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:23:02.072383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.072325 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:23:02.072383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.072357 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:23:02.424961 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.424883 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" event={"ID":"3eee358e-77c5-4fb9-b377-ecac27b13e0f","Type":"ContainerStarted","Data":"b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c"} Nov 03 01:23:02.424961 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.424948 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" event={"ID":"3eee358e-77c5-4fb9-b377-ecac27b13e0f","Type":"ContainerStarted","Data":"73c0e6bf1cbf6b6718b61e8e3f3be7a03e2ce22ce88705321761dcd8730ae655"} Nov 03 01:23:02.541708 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.541662 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podStartSLOduration=1.54161945 podStartE2EDuration="1.54161945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:02.540897168 +0000 UTC m=+4672.686591623" watchObservedRunningTime="2026-11-03 01:23:02.54161945 +0000 UTC m=+4672.687313906" Nov 03 01:23:02.641095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:02.641068 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:03.646365 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:03.646329 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:04.437187 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:04.437141 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:04.447325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:04.444889 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:08.258680 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.258649 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc"] Nov 03 01:23:08.259196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.259148 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e28b6f2-d0b2-4346-9a0b-e706d0ac3363" containerName="istio-proxy" Nov 03 01:23:08.262590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.262574 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.312416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.312383 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.312535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.312510 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbs97\" (UniqueName: \"kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97\") pod \"payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.317209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.317187 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc"] Nov 03 01:23:08.363732 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.363692 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-337498b9\"" Nov 03 01:23:08.363732 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.363725 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-337498b9-dockercfg-xd47q\"" Nov 03 01:23:08.413304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.413273 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbs97\" (UniqueName: \"kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97\") pod \"payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.413470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.413382 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.414068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.414050 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.473766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.473738 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbs97\" (UniqueName: \"kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97\") pod \"payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.576623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.576563 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:08.949893 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:08.949869 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc"] Nov 03 01:23:08.951394 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:23:08.951356 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afd77f3_6160_4b7d_a7a1_9c5e90723965.slice/crio-8655b418deacca7988d14fe836a2c017ed1509e355c41f7cf6bff25a1049e53f WatchSource:0}: Error finding container 8655b418deacca7988d14fe836a2c017ed1509e355c41f7cf6bff25a1049e53f: Status 404 returned error can't find the container with id 8655b418deacca7988d14fe836a2c017ed1509e355c41f7cf6bff25a1049e53f Nov 03 01:23:09.476646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:09.476599 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" event={"ID":"9afd77f3-6160-4b7d-a7a1-9c5e90723965","Type":"ContainerStarted","Data":"5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9"} Nov 03 01:23:09.477053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:09.476707 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:09.477053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:09.476734 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" event={"ID":"9afd77f3-6160-4b7d-a7a1-9c5e90723965","Type":"ContainerStarted","Data":"8655b418deacca7988d14fe836a2c017ed1509e355c41f7cf6bff25a1049e53f"} Nov 03 01:23:09.589783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:09.589736 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" podStartSLOduration=1.589723422 podStartE2EDuration="1.589723422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:09.587909805 +0000 UTC m=+4679.733604262" watchObservedRunningTime="2026-11-03 01:23:09.589723422 +0000 UTC m=+4679.735417878" Nov 03 01:23:12.155684 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:23:12.155649 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:23:14.211845 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:14.211814 5144 scope.go:122] "RemoveContainer" containerID="e7bf0d0bcf4222195f84b7ba46f8704fa28c56304b25207b9f5568792089b249" Nov 03 01:23:14.221871 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:14.221848 5144 scope.go:122] "RemoveContainer" containerID="1f7d6b748d2648449a3765b1492f47cd35a5a69ff89751982dd22716d9d6bafc" Nov 03 01:23:20.487017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:20.486987 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:27.596566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:27.596540 5144 generic.go:356] "Generic (PLEG): container finished" podID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerID="b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c" exitCode=0 Nov 03 01:23:27.596959 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:27.596621 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" event={"ID":"3eee358e-77c5-4fb9-b377-ecac27b13e0f","Type":"ContainerDied","Data":"b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c"} Nov 03 01:23:27.597016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:27.596980 5144 scope.go:122] "RemoveContainer" containerID="b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c" Nov 03 01:23:27.598200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:27.598172 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:23:27.598286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:27.598238 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:23:27.598286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:27.598272 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:23:28.604703 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:28.604672 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" event={"ID":"3eee358e-77c5-4fb9-b377-ecac27b13e0f","Type":"ContainerStarted","Data":"18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9"} Nov 03 01:23:28.641470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:28.641437 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:28.641672 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:28.641642 5144 patch_prober.go:29] interesting pod/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:23:28.641725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:28.641698 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:23:29.641978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:29.641942 5144 patch_prober.go:29] interesting pod/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:23:29.642436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:29.641993 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:23:30.641143 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:30.641105 5144 patch_prober.go:29] interesting pod/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:23:30.641374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:30.641172 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:23:31.136375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.136337 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8"] Nov 03 01:23:31.143316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.143287 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.163249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.163222 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.163359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.163253 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l67q4\" (UniqueName: \"kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.163359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.163273 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.173718 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.173695 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8"] Nov 03 01:23:31.236655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.227850 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:23:31.236655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.227927 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:23:31.264211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.264175 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.264336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.264213 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l67q4\" (UniqueName: \"kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.264336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.264257 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.264882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.264862 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.266832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.266804 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.339583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.339547 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l67q4\" (UniqueName: \"kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4\") pod \"maas-api-revoke-keys-e2e-ait-337498b9-7cqt8\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.455830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.455762 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:31.641778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.641738 5144 patch_prober.go:29] interesting pod/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:23:31.641951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.641801 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:23:31.841774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:31.841749 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8"] Nov 03 01:23:31.844058 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:23:31.844026 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c380b4_d230_4e41_a2a2_0956381ffe78.slice/crio-c4c2991e5cb8d910244e6819ec743e43ca7bc6e58a13cbf163c934022166e5e6 WatchSource:0}: Error finding container c4c2991e5cb8d910244e6819ec743e43ca7bc6e58a13cbf163c934022166e5e6: Status 404 returned error can't find the container with id c4c2991e5cb8d910244e6819ec743e43ca7bc6e58a13cbf163c934022166e5e6 Nov 03 01:23:32.641386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:32.641352 5144 patch_prober.go:29] interesting pod/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" start-of-body= Nov 03 01:23:32.641861 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:32.641404 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.100:15021/healthz/ready\": dial tcp 10.132.1.100:15021: connect: connection refused" Nov 03 01:23:32.653527 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:32.653497 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" event={"ID":"23c380b4-d230-4e41-a2a2-0956381ffe78","Type":"ContainerStarted","Data":"bf4e5a2ed73e050f7f9510f708bc2a3fd7b190b1114c2440ee06bdfd11c2bfeb"} Nov 03 01:23:32.653734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:32.653552 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" event={"ID":"23c380b4-d230-4e41-a2a2-0956381ffe78","Type":"ContainerStarted","Data":"c4c2991e5cb8d910244e6819ec743e43ca7bc6e58a13cbf163c934022166e5e6"} Nov 03 01:23:32.765255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:32.765204 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" podStartSLOduration=1.76518909 podStartE2EDuration="1.76518909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:32.764840432 +0000 UTC m=+4702.910534912" watchObservedRunningTime="2026-11-03 01:23:32.76518909 +0000 UTC m=+4702.910883546" Nov 03 01:23:33.644815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:33.644786 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:33.645218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:33.645200 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:33.645863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:33.645844 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:33.660615 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:33.660591 5144 generic.go:356] "Generic (PLEG): container finished" podID="23c380b4-d230-4e41-a2a2-0956381ffe78" containerID="bf4e5a2ed73e050f7f9510f708bc2a3fd7b190b1114c2440ee06bdfd11c2bfeb" exitCode=0 Nov 03 01:23:33.660746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:33.660674 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" event={"ID":"23c380b4-d230-4e41-a2a2-0956381ffe78","Type":"ContainerDied","Data":"bf4e5a2ed73e050f7f9510f708bc2a3fd7b190b1114c2440ee06bdfd11c2bfeb"} Nov 03 01:23:34.805350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.805326 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:34.900853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.900822 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4\" (UniqueName: \"kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4\") pod \"23c380b4-d230-4e41-a2a2-0956381ffe78\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " Nov 03 01:23:34.900997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.900895 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca\") pod \"23c380b4-d230-4e41-a2a2-0956381ffe78\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " Nov 03 01:23:34.900997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.900918 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca\") pod \"23c380b4-d230-4e41-a2a2-0956381ffe78\" (UID: \"23c380b4-d230-4e41-a2a2-0956381ffe78\") " Nov 03 01:23:34.901260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.901227 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca" pod "23c380b4-d230-4e41-a2a2-0956381ffe78" (UID: "23c380b4-d230-4e41-a2a2-0956381ffe78"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:34.903083 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.903061 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca" pod "23c380b4-d230-4e41-a2a2-0956381ffe78" (UID: "23c380b4-d230-4e41-a2a2-0956381ffe78"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:23:34.903171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:34.903120 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4" pod "23c380b4-d230-4e41-a2a2-0956381ffe78" (UID: "23c380b4-d230-4e41-a2a2-0956381ffe78"). InnerVolumeSpecName "kube-api-access-l67q4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:35.001881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:35.001807 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l67q4\" (UniqueName: \"kubernetes.io/projected/23c380b4-d230-4e41-a2a2-0956381ffe78-kube-api-access-l67q4\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:35.001881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:35.001835 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:35.001881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:35.001845 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23c380b4-d230-4e41-a2a2-0956381ffe78-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:35.673981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:35.673952 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" Nov 03 01:23:35.674157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:35.673989 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8" event={"ID":"23c380b4-d230-4e41-a2a2-0956381ffe78","Type":"ContainerDied","Data":"c4c2991e5cb8d910244e6819ec743e43ca7bc6e58a13cbf163c934022166e5e6"} Nov 03 01:23:35.674157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:35.674034 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4c2991e5cb8d910244e6819ec743e43ca7bc6e58a13cbf163c934022166e5e6" Nov 03 01:23:36.046694 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:36.046649 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt"] Nov 03 01:23:36.049600 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:36.049560 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6f28d171-999gt"] Nov 03 01:23:37.995257 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:37.995226 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="823add80-767e-465c-abbc-936177af2629" path="/var/lib/kubelet/pods/823add80-767e-465c-abbc-936177af2629/volumes" Nov 03 01:23:42.155741 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:23:42.155699 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:23:42.719438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:42.719408 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc"] Nov 03 01:23:47.719995 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:47.719955 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" podUID="9afd77f3-6160-4b7d-a7a1-9c5e90723965" containerName="payload-pre-processing" containerID="cri-o://5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9" gracePeriod=25 Nov 03 01:23:47.967166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:47.967143 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:48.017948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.017920 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume\") pod \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " Nov 03 01:23:48.018105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.018022 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97\" (UniqueName: \"kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97\") pod \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\" (UID: \"9afd77f3-6160-4b7d-a7a1-9c5e90723965\") " Nov 03 01:23:48.018302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.018279 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume" pod "9afd77f3-6160-4b7d-a7a1-9c5e90723965" (UID: "9afd77f3-6160-4b7d-a7a1-9c5e90723965"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:48.020174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.020149 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97" pod "9afd77f3-6160-4b7d-a7a1-9c5e90723965" (UID: "9afd77f3-6160-4b7d-a7a1-9c5e90723965"). InnerVolumeSpecName "kube-api-access-zbs97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:48.119111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.119085 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afd77f3-6160-4b7d-a7a1-9c5e90723965-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:48.119111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.119110 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbs97\" (UniqueName: \"kubernetes.io/projected/9afd77f3-6160-4b7d-a7a1-9c5e90723965-kube-api-access-zbs97\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:48.755031 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.754994 5144 generic.go:356] "Generic (PLEG): container finished" podID="9afd77f3-6160-4b7d-a7a1-9c5e90723965" containerID="5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9" exitCode=0 Nov 03 01:23:48.755495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.755046 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" Nov 03 01:23:48.755495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.755083 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" event={"ID":"9afd77f3-6160-4b7d-a7a1-9c5e90723965","Type":"ContainerDied","Data":"5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9"} Nov 03 01:23:48.755495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.755146 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc" event={"ID":"9afd77f3-6160-4b7d-a7a1-9c5e90723965","Type":"ContainerDied","Data":"8655b418deacca7988d14fe836a2c017ed1509e355c41f7cf6bff25a1049e53f"} Nov 03 01:23:48.755495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.755176 5144 scope.go:122] "RemoveContainer" containerID="5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9" Nov 03 01:23:48.765005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.764984 5144 scope.go:122] "RemoveContainer" containerID="5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9" Nov 03 01:23:48.765501 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:23:48.765291 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9\": container with ID starting with 5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9 not found: ID does not exist" containerID="5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9" Nov 03 01:23:48.765501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.765331 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9"} err="failed to get container status \"5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9\": rpc error: code = NotFound desc = could not find container \"5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9\": container with ID starting with 5b361961423ed0ba74164d73f81841af5371a4805b0ffe4864856538d0f937f9 not found: ID does not exist" Nov 03 01:23:48.920261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.920186 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc"] Nov 03 01:23:48.969472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:48.969441 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-337498b9-68c877c4f5-gx5xc"] Nov 03 01:23:49.045649 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:49.045573 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd"] Nov 03 01:23:49.048100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:49.048081 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7a79ba91-qzrpd"] Nov 03 01:23:49.995218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:49.995183 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9afd77f3-6160-4b7d-a7a1-9c5e90723965" path="/var/lib/kubelet/pods/9afd77f3-6160-4b7d-a7a1-9c5e90723965/volumes" Nov 03 01:23:49.995714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:49.995584 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd68a66-a0b8-4fdc-b63a-45ac2597deee" path="/var/lib/kubelet/pods/ffd68a66-a0b8-4fdc-b63a-45ac2597deee/volumes" Nov 03 01:23:50.763752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:50.763719 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8"] Nov 03 01:23:50.813179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:50.813152 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-337498b9-7cqt8"] Nov 03 01:23:51.995419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:51.995386 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c380b4-d230-4e41-a2a2-0956381ffe78" path="/var/lib/kubelet/pods/23c380b4-d230-4e41-a2a2-0956381ffe78/volumes" Nov 03 01:23:52.254686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:52.254663 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg"] Nov 03 01:23:52.254894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:52.254873 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" containerID="cri-o://18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9" gracePeriod=30 Nov 03 01:23:57.507190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.507163 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:57.599784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599727 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.599784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599760 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.599968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599851 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.599968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599871 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.599968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599907 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.599968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599927 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.600186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599959 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:57.600186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.599973 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.600186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600012 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.600186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600045 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy\") pod \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\" (UID: \"3eee358e-77c5-4fb9-b377-ecac27b13e0f\") " Nov 03 01:23:57.600186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600139 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600241 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600252 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600317 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600363 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600385 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600399 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.600461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.600413 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.601988 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.601961 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:57.602316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.602286 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:23:57.602605 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.602585 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:23:57.602730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.602710 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs" pod "3eee358e-77c5-4fb9-b377-ecac27b13e0f" (UID: "3eee358e-77c5-4fb9-b377-ecac27b13e0f"). InnerVolumeSpecName "kube-api-access-7kgcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:23:57.701077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.701035 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.701077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.701062 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.701077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.701078 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kgcs\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-kube-api-access-7kgcs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.701272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.701092 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.701272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.701105 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eee358e-77c5-4fb9-b377-ecac27b13e0f-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:23:57.814172 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.814141 5144 generic.go:356] "Generic (PLEG): container finished" podID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerID="18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9" exitCode=0 Nov 03 01:23:57.814348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.814190 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" Nov 03 01:23:57.814348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.814231 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" event={"ID":"3eee358e-77c5-4fb9-b377-ecac27b13e0f","Type":"ContainerDied","Data":"18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9"} Nov 03 01:23:57.814348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.814290 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg" event={"ID":"3eee358e-77c5-4fb9-b377-ecac27b13e0f","Type":"ContainerDied","Data":"73c0e6bf1cbf6b6718b61e8e3f3be7a03e2ce22ce88705321761dcd8730ae655"} Nov 03 01:23:57.814348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.814313 5144 scope.go:122] "RemoveContainer" containerID="18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9" Nov 03 01:23:57.824802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.824783 5144 scope.go:122] "RemoveContainer" containerID="b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c" Nov 03 01:23:57.833471 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.833456 5144 scope.go:122] "RemoveContainer" containerID="18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9" Nov 03 01:23:57.833707 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:23:57.833687 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9\": container with ID starting with 18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9 not found: ID does not exist" containerID="18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9" Nov 03 01:23:57.833769 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.833714 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9"} err="failed to get container status \"18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9\": rpc error: code = NotFound desc = could not find container \"18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9\": container with ID starting with 18e6ab4985a220d4d1ecf1741c0aefc6c0676236ca786a5464a249e32ade81a9 not found: ID does not exist" Nov 03 01:23:57.833769 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.833730 5144 scope.go:122] "RemoveContainer" containerID="b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c" Nov 03 01:23:57.833998 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:23:57.833981 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c\": container with ID starting with b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c not found: ID does not exist" containerID="b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c" Nov 03 01:23:57.834052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.834002 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c"} err="failed to get container status \"b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c\": rpc error: code = NotFound desc = could not find container \"b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c\": container with ID starting with b30e662ab68fb8974e0bcbed15ae50bb8ae688271573675d0f566298736c689c not found: ID does not exist" Nov 03 01:23:57.977220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.977156 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg"] Nov 03 01:23:57.979872 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.979853 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-337498b9-openshift-default-645db8f455-t2lcg"] Nov 03 01:23:57.994936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:57.994913 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" path="/var/lib/kubelet/pods/3eee358e-77c5-4fb9-b377-ecac27b13e0f/volumes" Nov 03 01:23:58.547583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.547548 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-785d4477d7-s2c6t"] Nov 03 01:23:58.548043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.547982 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" Nov 03 01:23:58.548043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.547994 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" Nov 03 01:23:58.548043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548011 5144 cpu_manager.go:414] "containerMap: removing container" podUID="23c380b4-d230-4e41-a2a2-0956381ffe78" containerName="revoke-keys" Nov 03 01:23:58.548043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548016 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c380b4-d230-4e41-a2a2-0956381ffe78" containerName="revoke-keys" Nov 03 01:23:58.548043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548042 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9afd77f3-6160-4b7d-a7a1-9c5e90723965" containerName="payload-pre-processing" Nov 03 01:23:58.548043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548048 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9afd77f3-6160-4b7d-a7a1-9c5e90723965" containerName="payload-pre-processing" Nov 03 01:23:58.548433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548116 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" Nov 03 01:23:58.548433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548124 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9afd77f3-6160-4b7d-a7a1-9c5e90723965" containerName="payload-pre-processing" Nov 03 01:23:58.548433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548130 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" Nov 03 01:23:58.548433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.548138 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c380b4-d230-4e41-a2a2-0956381ffe78" containerName="revoke-keys" Nov 03 01:23:58.551447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.551428 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.604030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.604006 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 01:23:58.604164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.604083 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 01:23:58.604164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.604129 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 01:23:58.604164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.604140 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rcpk7\"" Nov 03 01:23:58.609519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.609497 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.609611 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.609548 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.609684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.609662 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpkrn\" (UniqueName: \"kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.657824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.657802 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785d4477d7-s2c6t"] Nov 03 01:23:58.710230 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.710193 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.710375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.710248 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpkrn\" (UniqueName: \"kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.710375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.710353 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.712822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.712791 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.712990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.712970 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.767023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.767000 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpkrn\" (UniqueName: \"kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn\") pod \"maas-controller-785d4477d7-s2c6t\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:58.864532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:58.864471 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:23:59.234468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:59.234439 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785d4477d7-s2c6t"] Nov 03 01:23:59.239681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:59.239659 5144 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:23:59.829451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:23:59.829415 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-s2c6t" event={"ID":"7ab62c68-f102-4ba2-a9b3-231881383038","Type":"ContainerStarted","Data":"c539e4c3048da2a80c7d7d3e46c102ffaa8f66a08f7032ce4a24599764287a5a"} Nov 03 01:24:02.851146 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:02.851110 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-s2c6t" event={"ID":"7ab62c68-f102-4ba2-a9b3-231881383038","Type":"ContainerStarted","Data":"ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725"} Nov 03 01:24:02.851546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:02.851178 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:24:02.968932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:02.968870 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-785d4477d7-s2c6t" podStartSLOduration=1.7430697130000001 podStartE2EDuration="4.968852901s" totalImagesPullingTime="3.225783188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:02.966000263 +0000 UTC m=+4733.111694719" watchObservedRunningTime="2026-11-03 01:24:02.968852901 +0000 UTC m=+4733.114547358" Nov 03 01:24:09.046296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:09.046263 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn"] Nov 03 01:24:09.095668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:09.095623 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c162b859-qtrcn"] Nov 03 01:24:09.994804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:09.994770 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dfe161-0d32-467c-8fda-8aa78362b33f" path="/var/lib/kubelet/pods/f5dfe161-0d32-467c-8fda-8aa78362b33f/volumes" Nov 03 01:24:12.153826 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:24:12.153787 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:24:13.859969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:13.859941 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:24:14.264217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:14.264188 5144 scope.go:122] "RemoveContainer" containerID="0539a14369fec689b73776ac3297d355bebc520362b907e467d7af3cc2f0fbe2" Nov 03 01:24:14.274166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:14.274145 5144 scope.go:122] "RemoveContainer" containerID="7e0b1e821abf99e6e5a00f15e0921e2220647176ffb092ebd4117a8f3b2676db" Nov 03 01:24:19.044710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:19.044678 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9"] Nov 03 01:24:19.094139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:19.094115 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03f92fff-x8rx9"] Nov 03 01:24:19.994345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:19.994313 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3941e883-dd6c-40e4-911a-8c6f46455248" path="/var/lib/kubelet/pods/3941e883-dd6c-40e4-911a-8c6f46455248/volumes" Nov 03 01:24:23.004519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.004483 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp"] Nov 03 01:24:23.005000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.004894 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" Nov 03 01:24:23.005000 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.004906 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3eee358e-77c5-4fb9-b377-ecac27b13e0f" containerName="istio-proxy" Nov 03 01:24:23.008055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.008039 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.063387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.063360 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp"] Nov 03 01:24:23.109766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.109740 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:24:23.109914 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.109886 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:24:23.119049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.119020 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.119182 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.119062 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xclg\" (UniqueName: \"kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.119235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.119198 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.220112 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.220081 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.220317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.220156 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.220317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.220176 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xclg\" (UniqueName: \"kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.220848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.220827 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.222596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.222561 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.276828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.276769 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xclg\" (UniqueName: \"kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg\") pod \"maas-api-revoke-sub-85237e45-hq8xp\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.321687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.321663 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:23.695418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.695383 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp"] Nov 03 01:24:23.977183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:23.977118 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" event={"ID":"32fe01be-f261-4580-a8aa-475d700433b5","Type":"ContainerStarted","Data":"45b2629d322a85434ca3013b636677ef204ed7300a0f4c5375c204901113011a"} Nov 03 01:24:24.985772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:24.985696 5144 generic.go:356] "Generic (PLEG): container finished" podID="32fe01be-f261-4580-a8aa-475d700433b5" containerID="2fd42157ce1c2a8b8ecfa533855766bf6d16284249e95a53d51262567525b346" exitCode=0 Nov 03 01:24:24.985772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:24.985761 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" event={"ID":"32fe01be-f261-4580-a8aa-475d700433b5","Type":"ContainerDied","Data":"2fd42157ce1c2a8b8ecfa533855766bf6d16284249e95a53d51262567525b346"} Nov 03 01:24:25.091454 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.091427 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw"] Nov 03 01:24:25.095200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.095179 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.146277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.146245 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw"] Nov 03 01:24:25.148316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.148290 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zx4cj\"" Nov 03 01:24:25.240291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240216 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240248 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240292 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240346 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240444 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdgn6\" (UniqueName: \"kubernetes.io/projected/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-kube-api-access-xdgn6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240497 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240466 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240547 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240602 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.240750 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.240722 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341159 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341204 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341231 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341428 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341471 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341531 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdgn6\" (UniqueName: \"kubernetes.io/projected/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-kube-api-access-xdgn6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341566 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341625 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341598 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341604 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341799 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.341959 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.341717 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.342029 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.342001 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.342106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.342081 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.342334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.342313 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.343999 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.343975 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.344682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.344665 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.397032 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.397010 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.397460 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.397437 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdgn6\" (UniqueName: \"kubernetes.io/projected/43f74203-a9e7-4cfe-ab2a-596a0ea37e1c-kube-api-access-xdgn6\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw\" (UID: \"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.409199 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.409181 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:25.834914 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.834884 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw"] Nov 03 01:24:25.835251 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:24:25.835227 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43f74203_a9e7_4cfe_ab2a_596a0ea37e1c.slice/crio-35d466c107c22325c2879a4e6b9c8a31f73bd48076f819a0bc106c5fb4c4a1a3 WatchSource:0}: Error finding container 35d466c107c22325c2879a4e6b9c8a31f73bd48076f819a0bc106c5fb4c4a1a3: Status 404 returned error can't find the container with id 35d466c107c22325c2879a4e6b9c8a31f73bd48076f819a0bc106c5fb4c4a1a3 Nov 03 01:24:25.837428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.837394 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:25.837515 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.837469 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:25.837569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.837513 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:25.996978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.996944 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" event={"ID":"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c","Type":"ContainerStarted","Data":"87aac94e5052272bf56dad98265e664d5243059b2191dfe14ab8835baee352d6"} Nov 03 01:24:25.997443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:25.996999 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" event={"ID":"43f74203-a9e7-4cfe-ab2a-596a0ea37e1c","Type":"ContainerStarted","Data":"35d466c107c22325c2879a4e6b9c8a31f73bd48076f819a0bc106c5fb4c4a1a3"} Nov 03 01:24:26.112186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.112098 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podStartSLOduration=2.112082746 podStartE2EDuration="2.112082746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:26.108762804 +0000 UTC m=+4756.254457260" watchObservedRunningTime="2026-11-03 01:24:26.112082746 +0000 UTC m=+4756.257777192" Nov 03 01:24:26.122423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.122400 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:26.147570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.147548 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca\") pod \"32fe01be-f261-4580-a8aa-475d700433b5\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " Nov 03 01:24:26.147697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.147672 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg\" (UniqueName: \"kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg\") pod \"32fe01be-f261-4580-a8aa-475d700433b5\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " Nov 03 01:24:26.147745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.147724 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca\") pod \"32fe01be-f261-4580-a8aa-475d700433b5\" (UID: \"32fe01be-f261-4580-a8aa-475d700433b5\") " Nov 03 01:24:26.147778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.147761 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca" pod "32fe01be-f261-4580-a8aa-475d700433b5" (UID: "32fe01be-f261-4580-a8aa-475d700433b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:24:26.148039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.148019 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:24:26.150154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.150133 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca" pod "32fe01be-f261-4580-a8aa-475d700433b5" (UID: "32fe01be-f261-4580-a8aa-475d700433b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:26.150356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.150327 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg" pod "32fe01be-f261-4580-a8aa-475d700433b5" (UID: "32fe01be-f261-4580-a8aa-475d700433b5"). InnerVolumeSpecName "kube-api-access-7xclg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:26.248689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.248652 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32fe01be-f261-4580-a8aa-475d700433b5-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:24:26.248689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.248691 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xclg\" (UniqueName: \"kubernetes.io/projected/32fe01be-f261-4580-a8aa-475d700433b5-kube-api-access-7xclg\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:24:26.410101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.410037 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:26.411082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.411058 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:26.411154 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:26.411124 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:27.003108 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:27.003075 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" Nov 03 01:24:27.003532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:27.003118 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp" event={"ID":"32fe01be-f261-4580-a8aa-475d700433b5","Type":"ContainerDied","Data":"45b2629d322a85434ca3013b636677ef204ed7300a0f4c5375c204901113011a"} Nov 03 01:24:27.003532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:27.003165 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45b2629d322a85434ca3013b636677ef204ed7300a0f4c5375c204901113011a" Nov 03 01:24:27.410302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:27.410224 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:27.410302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:27.410278 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:28.410584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:28.410542 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:28.411022 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:28.410608 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:29.409876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:29.409842 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:29.410063 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:29.409894 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:30.409643 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:30.409585 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:30.410181 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:30.409650 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:31.409923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:31.409891 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:31.410310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:31.409945 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:32.410046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:32.409998 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" start-of-body= Nov 03 01:24:32.410591 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:32.410072 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": dial tcp 10.132.1.105:15021: connect: connection refused" Nov 03 01:24:34.409768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:34.409723 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:24:34.410193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:34.409777 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:35.450833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.450777 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:24:35.451379 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.450851 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:24:35.611406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.611377 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-785d4477d7-s2c6t"] Nov 03 01:24:35.611592 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.611563 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-785d4477d7-s2c6t" podUID="7ab62c68-f102-4ba2-a9b3-231881383038" containerName="manager" containerID="cri-o://ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725" gracePeriod=30 Nov 03 01:24:35.884283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.884260 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:24:35.935643 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.935604 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn\" (UniqueName: \"kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn\") pod \"7ab62c68-f102-4ba2-a9b3-231881383038\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " Nov 03 01:24:35.935779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.935693 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert\") pod \"7ab62c68-f102-4ba2-a9b3-231881383038\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " Nov 03 01:24:35.935779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.935719 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs\") pod \"7ab62c68-f102-4ba2-a9b3-231881383038\" (UID: \"7ab62c68-f102-4ba2-a9b3-231881383038\") " Nov 03 01:24:35.937845 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.937816 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn" pod "7ab62c68-f102-4ba2-a9b3-231881383038" (UID: "7ab62c68-f102-4ba2-a9b3-231881383038"). InnerVolumeSpecName "kube-api-access-tpkrn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:24:35.937990 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.937912 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert" pod "7ab62c68-f102-4ba2-a9b3-231881383038" (UID: "7ab62c68-f102-4ba2-a9b3-231881383038"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:35.938065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.938046 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs" pod "7ab62c68-f102-4ba2-a9b3-231881383038" (UID: "7ab62c68-f102-4ba2-a9b3-231881383038"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:24:35.944099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.944077 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:35.944441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.944422 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:35.945478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:35.945426 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" Nov 03 01:24:36.037336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.037251 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpkrn\" (UniqueName: \"kubernetes.io/projected/7ab62c68-f102-4ba2-a9b3-231881383038-kube-api-access-tpkrn\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:24:36.037336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.037284 5144 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-webhook-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:24:36.037336 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.037297 5144 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ab62c68-f102-4ba2-a9b3-231881383038-metrics-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:24:36.080076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.080045 5144 generic.go:356] "Generic (PLEG): container finished" podID="7ab62c68-f102-4ba2-a9b3-231881383038" containerID="ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725" exitCode=0 Nov 03 01:24:36.080205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.080090 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-s2c6t" Nov 03 01:24:36.080205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.080121 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-s2c6t" event={"ID":"7ab62c68-f102-4ba2-a9b3-231881383038","Type":"ContainerDied","Data":"ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725"} Nov 03 01:24:36.080205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.080173 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-s2c6t" event={"ID":"7ab62c68-f102-4ba2-a9b3-231881383038","Type":"ContainerDied","Data":"c539e4c3048da2a80c7d7d3e46c102ffaa8f66a08f7032ce4a24599764287a5a"} Nov 03 01:24:36.080205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.080196 5144 scope.go:122] "RemoveContainer" containerID="ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725" Nov 03 01:24:36.089596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.089576 5144 scope.go:122] "RemoveContainer" containerID="ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725" Nov 03 01:24:36.089904 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:24:36.089881 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725\": container with ID starting with ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725 not found: ID does not exist" containerID="ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725" Nov 03 01:24:36.089994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.089913 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725"} err="failed to get container status \"ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725\": rpc error: code = NotFound desc = could not find container \"ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725\": container with ID starting with ad3709833bb5e451065ee63f2f38447874c32f80dab123c36fb745cc8e9f9725 not found: ID does not exist" Nov 03 01:24:36.225863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.225834 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-785d4477d7-s2c6t"] Nov 03 01:24:36.230452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:36.230426 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-785d4477d7-s2c6t"] Nov 03 01:24:37.629665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.629622 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-785d4477d7-2j7w9"] Nov 03 01:24:37.630098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.630083 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7ab62c68-f102-4ba2-a9b3-231881383038" containerName="manager" Nov 03 01:24:37.630165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.630099 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab62c68-f102-4ba2-a9b3-231881383038" containerName="manager" Nov 03 01:24:37.630165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.630123 5144 cpu_manager.go:414] "containerMap: removing container" podUID="32fe01be-f261-4580-a8aa-475d700433b5" containerName="revoke-keys" Nov 03 01:24:37.630165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.630128 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32fe01be-f261-4580-a8aa-475d700433b5" containerName="revoke-keys" Nov 03 01:24:37.630263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.630220 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab62c68-f102-4ba2-a9b3-231881383038" containerName="manager" Nov 03 01:24:37.630263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.630233 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="32fe01be-f261-4580-a8aa-475d700433b5" containerName="revoke-keys" Nov 03 01:24:37.633310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.633295 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.652209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.652189 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8843a38e-7855-4a1c-88c1-113c9c723ff5-metrics-certs\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.652341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.652230 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8843a38e-7855-4a1c-88c1-113c9c723ff5-webhook-cert\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.652341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.652299 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64775\" (UniqueName: \"kubernetes.io/projected/8843a38e-7855-4a1c-88c1-113c9c723ff5-kube-api-access-64775\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.685488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.685463 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 01:24:37.734712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.734690 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-rcpk7\"" Nov 03 01:24:37.734712 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.734707 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 01:24:37.734947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.734929 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 01:24:37.738482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.738462 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785d4477d7-2j7w9"] Nov 03 01:24:37.753281 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.753246 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8843a38e-7855-4a1c-88c1-113c9c723ff5-metrics-certs\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.753420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.753317 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8843a38e-7855-4a1c-88c1-113c9c723ff5-webhook-cert\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.753420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.753376 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64775\" (UniqueName: \"kubernetes.io/projected/8843a38e-7855-4a1c-88c1-113c9c723ff5-kube-api-access-64775\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.756044 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.756021 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8843a38e-7855-4a1c-88c1-113c9c723ff5-metrics-certs\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.756141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.756113 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8843a38e-7855-4a1c-88c1-113c9c723ff5-webhook-cert\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.809691 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.809667 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64775\" (UniqueName: \"kubernetes.io/projected/8843a38e-7855-4a1c-88c1-113c9c723ff5-kube-api-access-64775\") pod \"maas-controller-785d4477d7-2j7w9\" (UID: \"8843a38e-7855-4a1c-88c1-113c9c723ff5\") " pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.945901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.945845 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:37.996404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:37.996367 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab62c68-f102-4ba2-a9b3-231881383038" path="/var/lib/kubelet/pods/7ab62c68-f102-4ba2-a9b3-231881383038/volumes" Nov 03 01:24:38.315885 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:38.315852 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-785d4477d7-2j7w9"] Nov 03 01:24:38.316688 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:24:38.316664 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8843a38e_7855_4a1c_88c1_113c9c723ff5.slice/crio-4b82a4868d93aef3f5b9f23c4212f8afbb71ee599d1e59f84992b3cedaab3c08 WatchSource:0}: Error finding container 4b82a4868d93aef3f5b9f23c4212f8afbb71ee599d1e59f84992b3cedaab3c08: Status 404 returned error can't find the container with id 4b82a4868d93aef3f5b9f23c4212f8afbb71ee599d1e59f84992b3cedaab3c08 Nov 03 01:24:39.100367 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:39.100335 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-2j7w9" event={"ID":"8843a38e-7855-4a1c-88c1-113c9c723ff5","Type":"ContainerStarted","Data":"4b82a4868d93aef3f5b9f23c4212f8afbb71ee599d1e59f84992b3cedaab3c08"} Nov 03 01:24:40.108235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.108197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-785d4477d7-2j7w9" event={"ID":"8843a38e-7855-4a1c-88c1-113c9c723ff5","Type":"ContainerStarted","Data":"f57f6082f37fae6d79ed9ffdd3ce67f8154eee068754f7340fd76e21e8623fc5"} Nov 03 01:24:40.108617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.108315 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:40.152423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.152393 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq"] Nov 03 01:24:40.156049 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.156034 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.175913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.175891 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8z8z\" (UniqueName: \"kubernetes.io/projected/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-kube-api-access-g8z8z\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.175927 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.175961 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.175978 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.176167 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.176305 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.176379 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176456 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.176403 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.176498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.176476 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.257584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.257556 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq"] Nov 03 01:24:40.263018 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.262963 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-785d4477d7-2j7w9" podStartSLOduration=2.378317576 podStartE2EDuration="3.262945629s" totalImagesPullingTime="884.628053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:40.260968726 +0000 UTC m=+4770.406663205" watchObservedRunningTime="2026-11-03 01:24:40.262945629 +0000 UTC m=+4770.408640088" Nov 03 01:24:40.277126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277101 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277130 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277158 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277183 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277216 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277335 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277425 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277456 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8z8z\" (UniqueName: \"kubernetes.io/projected/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-kube-api-access-g8z8z\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277504 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277761 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277578 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.277949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.277927 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.278052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.278030 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.278052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.278043 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.278183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.278162 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.280425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.280389 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.280727 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.280706 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.333460 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.333431 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.333575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.333492 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8z8z\" (UniqueName: \"kubernetes.io/projected/9ea14538-faad-45d7-b7b8-8ac4e0362b0d-kube-api-access-g8z8z\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq\" (UID: \"9ea14538-faad-45d7-b7b8-8ac4e0362b0d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.468801 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.468729 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:40.890885 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.890849 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq"] Nov 03 01:24:40.891581 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:24:40.891560 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ea14538_faad_45d7_b7b8_8ac4e0362b0d.slice/crio-444c940fd5fca5a1936a33949062746c880237f619989f72f8c3998369aea08d WatchSource:0}: Error finding container 444c940fd5fca5a1936a33949062746c880237f619989f72f8c3998369aea08d: Status 404 returned error can't find the container with id 444c940fd5fca5a1936a33949062746c880237f619989f72f8c3998369aea08d Nov 03 01:24:40.893731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.893702 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:40.893796 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.893762 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:40.893796 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:40.893788 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:24:41.117371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:41.117334 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" event={"ID":"9ea14538-faad-45d7-b7b8-8ac4e0362b0d","Type":"ContainerStarted","Data":"a4eac58fa202797fa6bb4716ed29b6725fe7ee57e05bf9f27c05cd647f5641fa"} Nov 03 01:24:41.117877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:41.117389 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" event={"ID":"9ea14538-faad-45d7-b7b8-8ac4e0362b0d","Type":"ContainerStarted","Data":"444c940fd5fca5a1936a33949062746c880237f619989f72f8c3998369aea08d"} Nov 03 01:24:41.235258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:41.235147 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podStartSLOduration=1.235131756 podStartE2EDuration="1.235131756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:41.231398771 +0000 UTC m=+4771.377093229" watchObservedRunningTime="2026-11-03 01:24:41.235131756 +0000 UTC m=+4771.380826212" Nov 03 01:24:41.469703 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:41.469673 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:41.471373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:41.471351 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:41.471520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:41.471414 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:42.166736 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:24:42.166700 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:24:42.469765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:42.469682 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:42.469765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:42.469737 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:43.469304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:43.469269 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:43.469726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:43.469335 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:44.469348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:44.469309 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:44.469754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:44.469380 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:45.469890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:45.469850 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:45.470400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:45.469930 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:46.469347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:46.469309 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:46.469531 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:46.469373 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:47.469366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:47.469330 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" start-of-body= Nov 03 01:24:47.469778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:47.469379 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": dial tcp 10.132.1.107:15021: connect: connection refused" Nov 03 01:24:48.989388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:48.989356 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:48.989912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:48.989816 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:48.990542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:48.990522 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" Nov 03 01:24:51.120090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:51.120059 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-785d4477d7-2j7w9" Nov 03 01:24:52.045102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:52.045070 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4"] Nov 03 01:24:52.094524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:52.094489 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af0e65ce-qh2b4"] Nov 03 01:24:53.996318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:53.996285 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38720dd6-b1e2-4108-9b3e-b59ff45f68d6" path="/var/lib/kubelet/pods/38720dd6-b1e2-4108-9b3e-b59ff45f68d6/volumes" Nov 03 01:24:54.089411 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:54.089382 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l"] Nov 03 01:24:54.091929 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:54.091908 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-725b559c-lt98l"] Nov 03 01:24:56.002509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:24:56.002474 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="948b3157-b4f3-4486-a016-4322047d4b88" path="/var/lib/kubelet/pods/948b3157-b4f3-4486-a016-4322047d4b88/volumes" Nov 03 01:25:05.054566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:05.054529 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj"] Nov 03 01:25:05.101063 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:05.101024 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-06ee0c2f-h9wxj"] Nov 03 01:25:05.996211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:05.996176 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16362872-a310-4406-8534-117656535136" path="/var/lib/kubelet/pods/16362872-a310-4406-8534-117656535136/volumes" Nov 03 01:25:08.045844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.045806 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx"] Nov 03 01:25:08.095242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.095210 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a3112e3-ckbgx"] Nov 03 01:25:08.205557 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.205524 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8"] Nov 03 01:25:08.209359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.209343 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.261805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.261773 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:25:08.261975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.261933 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:25:08.265129 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.265097 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8"] Nov 03 01:25:08.268498 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.268475 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.268600 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.268536 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.268686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.268655 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpd79\" (UniqueName: \"kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.369369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.369294 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.369369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.369346 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.369650 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.369375 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpd79\" (UniqueName: \"kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.370042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.370014 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.371925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.371905 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.425494 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.425462 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpd79\" (UniqueName: \"kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79\") pod \"maas-api-revoke-sub-f904bff9-47wp8\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.522835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.522804 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:08.899452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:08.899418 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8"] Nov 03 01:25:08.900805 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:25:08.900776 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod122c7943_a9e7_4bf0_b84f_bd3d1fd0d247.slice/crio-b312e7e7137aa5c2e8423036c2b26e01d8344d930bb4b21a7d5180cd5ce68223 WatchSource:0}: Error finding container b312e7e7137aa5c2e8423036c2b26e01d8344d930bb4b21a7d5180cd5ce68223: Status 404 returned error can't find the container with id b312e7e7137aa5c2e8423036c2b26e01d8344d930bb4b21a7d5180cd5ce68223 Nov 03 01:25:09.315877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:09.315845 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" event={"ID":"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247","Type":"ContainerStarted","Data":"723a6992a56d7e60821462dbce2d59b2d19b4b94c8197a68c36cb0f2079c076a"} Nov 03 01:25:09.316234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:09.315898 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" event={"ID":"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247","Type":"ContainerStarted","Data":"b312e7e7137aa5c2e8423036c2b26e01d8344d930bb4b21a7d5180cd5ce68223"} Nov 03 01:25:09.432066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:09.432020 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" podStartSLOduration=1.432005849 podStartE2EDuration="1.432005849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:09.430238975 +0000 UTC m=+4799.575933445" watchObservedRunningTime="2026-11-03 01:25:09.432005849 +0000 UTC m=+4799.577700318" Nov 03 01:25:09.996774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:09.996745 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e586190-7c6e-47f8-aed5-96bb5607cef9" path="/var/lib/kubelet/pods/7e586190-7c6e-47f8-aed5-96bb5607cef9/volumes" Nov 03 01:25:10.324318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:10.324284 5144 generic.go:356] "Generic (PLEG): container finished" podID="122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" containerID="723a6992a56d7e60821462dbce2d59b2d19b4b94c8197a68c36cb0f2079c076a" exitCode=0 Nov 03 01:25:10.324743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:10.324371 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" event={"ID":"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247","Type":"ContainerDied","Data":"723a6992a56d7e60821462dbce2d59b2d19b4b94c8197a68c36cb0f2079c076a"} Nov 03 01:25:11.169376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.169348 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:25:11.179987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.179966 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:25:11.466590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.466567 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:11.501408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.501376 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca\") pod \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " Nov 03 01:25:11.501538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.501427 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca\") pod \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " Nov 03 01:25:11.501612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.501574 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79\" (UniqueName: \"kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79\") pod \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\" (UID: \"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247\") " Nov 03 01:25:11.501792 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.501765 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca" pod "122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" (UID: "122c7943-a9e7-4bf0-b84f-bd3d1fd0d247"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:11.501936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.501918 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:25:11.503538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.503514 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca" pod "122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" (UID: "122c7943-a9e7-4bf0-b84f-bd3d1fd0d247"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:11.503651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.503613 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79" pod "122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" (UID: "122c7943-a9e7-4bf0-b84f-bd3d1fd0d247"). InnerVolumeSpecName "kube-api-access-lpd79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:11.602777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.602746 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:25:11.602777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:11.602774 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpd79\" (UniqueName: \"kubernetes.io/projected/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247-kube-api-access-lpd79\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:25:12.154785 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:25:12.154743 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:25:12.337577 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:12.337549 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" Nov 03 01:25:12.337783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:12.337593 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8" event={"ID":"122c7943-a9e7-4bf0-b84f-bd3d1fd0d247","Type":"ContainerDied","Data":"b312e7e7137aa5c2e8423036c2b26e01d8344d930bb4b21a7d5180cd5ce68223"} Nov 03 01:25:12.337783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:12.337668 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b312e7e7137aa5c2e8423036c2b26e01d8344d930bb4b21a7d5180cd5ce68223" Nov 03 01:25:14.318930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:14.318897 5144 scope.go:122] "RemoveContainer" containerID="f005538ee76375fdd5d49886ce3759623b6b2e90c44092fd11b64d0cbc6834cb" Nov 03 01:25:14.329082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:14.329058 5144 scope.go:122] "RemoveContainer" containerID="dc8f3c4dd456a723086e5551692a185be4146a0d66962940b10d7194a30fbf21" Nov 03 01:25:14.337903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:14.337884 5144 scope.go:122] "RemoveContainer" containerID="4164c20a887b2d27a0f4335e2f6d158ef73bcb21c0fd1adc64c56b673b7756f6" Nov 03 01:25:17.045065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:17.045030 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc"] Nov 03 01:25:17.094473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:17.094441 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3430b4b-vxxnc"] Nov 03 01:25:17.995653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:17.995604 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddd8fce-395f-4776-9988-b67eaa4a24f1" path="/var/lib/kubelet/pods/0ddd8fce-395f-4776-9988-b67eaa4a24f1/volumes" Nov 03 01:25:25.504451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.504375 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs"] Nov 03 01:25:25.504867 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.504851 5144 cpu_manager.go:414] "containerMap: removing container" podUID="122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" containerName="revoke-keys" Nov 03 01:25:25.504867 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.504863 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" containerName="revoke-keys" Nov 03 01:25:25.504951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.504933 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" containerName="revoke-keys" Nov 03 01:25:25.508001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.507985 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643346 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643309 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lw8w\" (UniqueName: \"kubernetes.io/projected/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-kube-api-access-2lw8w\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643362 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643454 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643502 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643611 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643710 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643741 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643799 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.643928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.643892 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.708677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.708623 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs"] Nov 03 01:25:25.744818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.744781 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.744818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.744817 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.744835 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.744963 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745031 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lw8w\" (UniqueName: \"kubernetes.io/projected/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-kube-api-access-2lw8w\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745064 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745133 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745186 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745229 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-credential-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745245 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745332 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-data\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745472 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-workload-certs\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.745542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.745512 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-workload-socket\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.746023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.746001 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.747589 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.747566 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-envoy\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.747824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.747805 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.801805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.801741 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-istio-token\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.801926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.801907 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lw8w\" (UniqueName: \"kubernetes.io/projected/c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14-kube-api-access-2lw8w\") pod \"maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs\" (UID: \"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:25.821947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:25.821921 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:26.246875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.246849 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs"] Nov 03 01:25:26.249735 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:25:26.249708 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4ebc0de_d5b0_4c2c_b4af_bdd212eb8c14.slice/crio-0cb6146c85e23068c285af46a841a32af4c61ef1b9af26dcfab3a3cc274bf122 WatchSource:0}: Error finding container 0cb6146c85e23068c285af46a841a32af4c61ef1b9af26dcfab3a3cc274bf122: Status 404 returned error can't find the container with id 0cb6146c85e23068c285af46a841a32af4c61ef1b9af26dcfab3a3cc274bf122 Nov 03 01:25:26.252721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.252683 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.252835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.252768 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.252835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.252814 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:25:26.428997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.428959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" event={"ID":"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14","Type":"ContainerStarted","Data":"34fd56dbe6a9653c86f9bb66a8d248df26f5cfd7d6d6c5658fd72177e1fba0b2"} Nov 03 01:25:26.429213 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.429019 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" event={"ID":"c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14","Type":"ContainerStarted","Data":"0cb6146c85e23068c285af46a841a32af4c61ef1b9af26dcfab3a3cc274bf122"} Nov 03 01:25:26.547333 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.547274 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podStartSLOduration=1.547260777 podStartE2EDuration="1.547260777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:26.545009674 +0000 UTC m=+4816.690704148" watchObservedRunningTime="2026-11-03 01:25:26.547260777 +0000 UTC m=+4816.692955234" Nov 03 01:25:26.824717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:26.822492 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:27.825221 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:27.825181 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 01:25:27.825957 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:27.825923 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 01:25:28.071656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:28.059795 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq"] Nov 03 01:25:28.097434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:28.097346 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c812d30b-bkxjq"] Nov 03 01:25:28.826040 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:28.825994 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 01:25:28.826594 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:28.826060 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 01:25:29.831038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:29.831000 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 01:25:29.831685 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:29.831653 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 01:25:29.996015 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:29.995985 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39f9c304-c69f-45a3-b5d0-2e605438984a" path="/var/lib/kubelet/pods/39f9c304-c69f-45a3-b5d0-2e605438984a/volumes" Nov 03 01:25:30.829781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:30.829741 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 01:25:30.830089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:30.830061 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 01:25:31.824091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:31.824049 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 01:25:31.824745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:31.824708 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 01:25:32.822837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:32.822799 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" start-of-body= Nov 03 01:25:32.823068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:32.822873 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": dial tcp 10.132.1.109:15021: connect: connection refused" Nov 03 01:25:34.486324 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:34.486296 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:34.486852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:34.486777 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:34.487589 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:34.487567 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" Nov 03 01:25:42.154488 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:25:42.154449 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:25:42.700244 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.700213 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v"] Nov 03 01:25:42.705198 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.705181 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.760860 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.760834 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v"] Nov 03 01:25:42.806615 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.806586 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:25:42.806767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.806706 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:25:42.826742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.826717 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp6cw\" (UniqueName: \"kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.826879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.826845 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.826941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.826916 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.928017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.927980 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.928180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.928057 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp6cw\" (UniqueName: \"kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.928180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.928123 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.928752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.928728 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.930597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.930576 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:42.984391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:42.984337 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp6cw\" (UniqueName: \"kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw\") pod \"maas-api-revoke-sub-f03629db-hjs8v\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:43.020446 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:43.020427 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:43.392114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:43.392090 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v"] Nov 03 01:25:43.393619 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:25:43.393593 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bd80da7_eb07_45a5_bc94_b131abb474b1.slice/crio-a6d10a3c996314e9647c6adda1195cf097fce4d1b9d883c221fa73608ec7750e WatchSource:0}: Error finding container a6d10a3c996314e9647c6adda1195cf097fce4d1b9d883c221fa73608ec7750e: Status 404 returned error can't find the container with id a6d10a3c996314e9647c6adda1195cf097fce4d1b9d883c221fa73608ec7750e Nov 03 01:25:43.625120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:43.625089 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" event={"ID":"1bd80da7-eb07-45a5-bc94-b131abb474b1","Type":"ContainerStarted","Data":"a6d10a3c996314e9647c6adda1195cf097fce4d1b9d883c221fa73608ec7750e"} Nov 03 01:25:44.633326 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:44.633246 5144 generic.go:356] "Generic (PLEG): container finished" podID="1bd80da7-eb07-45a5-bc94-b131abb474b1" containerID="219ee53971c905a3007e80d7b638849e21b88618e9b9f317e943bd674ea011a5" exitCode=0 Nov 03 01:25:44.633739 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:44.633342 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" event={"ID":"1bd80da7-eb07-45a5-bc94-b131abb474b1","Type":"ContainerDied","Data":"219ee53971c905a3007e80d7b638849e21b88618e9b9f317e943bd674ea011a5"} Nov 03 01:25:45.790991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.790963 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:45.856144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.856117 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca\") pod \"1bd80da7-eb07-45a5-bc94-b131abb474b1\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " Nov 03 01:25:45.856273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.856157 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca\") pod \"1bd80da7-eb07-45a5-bc94-b131abb474b1\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " Nov 03 01:25:45.856273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.856197 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw\" (UniqueName: \"kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw\") pod \"1bd80da7-eb07-45a5-bc94-b131abb474b1\" (UID: \"1bd80da7-eb07-45a5-bc94-b131abb474b1\") " Nov 03 01:25:45.856524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.856503 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca" pod "1bd80da7-eb07-45a5-bc94-b131abb474b1" (UID: "1bd80da7-eb07-45a5-bc94-b131abb474b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:25:45.856684 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.856667 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:25:45.858356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.858324 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw" pod "1bd80da7-eb07-45a5-bc94-b131abb474b1" (UID: "1bd80da7-eb07-45a5-bc94-b131abb474b1"). InnerVolumeSpecName "kube-api-access-wp6cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:25:45.858476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.858355 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca" pod "1bd80da7-eb07-45a5-bc94-b131abb474b1" (UID: "1bd80da7-eb07-45a5-bc94-b131abb474b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:25:45.957979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.957889 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bd80da7-eb07-45a5-bc94-b131abb474b1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:25:45.957979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:45.957925 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp6cw\" (UniqueName: \"kubernetes.io/projected/1bd80da7-eb07-45a5-bc94-b131abb474b1-kube-api-access-wp6cw\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:25:46.648686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:46.648654 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" Nov 03 01:25:46.648686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:46.648668 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v" event={"ID":"1bd80da7-eb07-45a5-bc94-b131abb474b1","Type":"ContainerDied","Data":"a6d10a3c996314e9647c6adda1195cf097fce4d1b9d883c221fa73608ec7750e"} Nov 03 01:25:46.648906 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:25:46.648737 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6d10a3c996314e9647c6adda1195cf097fce4d1b9d883c221fa73608ec7750e" Nov 03 01:26:03.672518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.672479 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk"] Nov 03 01:26:03.673231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.673210 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1bd80da7-eb07-45a5-bc94-b131abb474b1" containerName="revoke-keys" Nov 03 01:26:03.673294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.673236 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bd80da7-eb07-45a5-bc94-b131abb474b1" containerName="revoke-keys" Nov 03 01:26:03.673391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.673380 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bd80da7-eb07-45a5-bc94-b131abb474b1" containerName="revoke-keys" Nov 03 01:26:03.677349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.677323 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.731013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.730983 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk"] Nov 03 01:26:03.755580 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.755544 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.755778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.755727 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.755778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.755763 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95n68\" (UniqueName: \"kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.776321 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.776285 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:26:03.776321 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.776314 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:26:03.857169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.857134 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.857169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.857181 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95n68\" (UniqueName: \"kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.857476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.857214 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.857945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.857919 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.859945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.859919 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.914310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.914284 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95n68\" (UniqueName: \"kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68\") pod \"maas-api-revoke-sub-0165d7d2-7w2qk\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:03.993407 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:03.993329 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:04.363864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:04.363831 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk"] Nov 03 01:26:04.364622 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:26:04.364588 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f3c8be_f10c_4194_8acf_4b12806660d1.slice/crio-ae38f86bbb9ec2499b4174f64b702a4a3ed2dd06d3fc883b2a96c0c9783bb07d WatchSource:0}: Error finding container ae38f86bbb9ec2499b4174f64b702a4a3ed2dd06d3fc883b2a96c0c9783bb07d: Status 404 returned error can't find the container with id ae38f86bbb9ec2499b4174f64b702a4a3ed2dd06d3fc883b2a96c0c9783bb07d Nov 03 01:26:04.807345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:04.807301 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" event={"ID":"45f3c8be-f10c-4194-8acf-4b12806660d1","Type":"ContainerStarted","Data":"f2f8d7556a3e7671d10af06216d0bfe20a02c91f9fe811f8cd493b398a9b9743"} Nov 03 01:26:04.807864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:04.807367 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" event={"ID":"45f3c8be-f10c-4194-8acf-4b12806660d1","Type":"ContainerStarted","Data":"ae38f86bbb9ec2499b4174f64b702a4a3ed2dd06d3fc883b2a96c0c9783bb07d"} Nov 03 01:26:04.921724 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:04.921598 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" podStartSLOduration=1.921583516 podStartE2EDuration="1.921583516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:04.919565587 +0000 UTC m=+4855.065260041" watchObservedRunningTime="2026-11-03 01:26:04.921583516 +0000 UTC m=+4855.067277971" Nov 03 01:26:05.815979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:05.815938 5144 generic.go:356] "Generic (PLEG): container finished" podID="45f3c8be-f10c-4194-8acf-4b12806660d1" containerID="f2f8d7556a3e7671d10af06216d0bfe20a02c91f9fe811f8cd493b398a9b9743" exitCode=0 Nov 03 01:26:05.816415 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:05.816003 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" event={"ID":"45f3c8be-f10c-4194-8acf-4b12806660d1","Type":"ContainerDied","Data":"f2f8d7556a3e7671d10af06216d0bfe20a02c91f9fe811f8cd493b398a9b9743"} Nov 03 01:26:06.972345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:06.972319 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:07.087834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.087754 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68\" (UniqueName: \"kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68\") pod \"45f3c8be-f10c-4194-8acf-4b12806660d1\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " Nov 03 01:26:07.088042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.087838 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca\") pod \"45f3c8be-f10c-4194-8acf-4b12806660d1\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " Nov 03 01:26:07.088042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.087897 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca\") pod \"45f3c8be-f10c-4194-8acf-4b12806660d1\" (UID: \"45f3c8be-f10c-4194-8acf-4b12806660d1\") " Nov 03 01:26:07.088354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.088320 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca" pod "45f3c8be-f10c-4194-8acf-4b12806660d1" (UID: "45f3c8be-f10c-4194-8acf-4b12806660d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:07.090179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.090148 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68" pod "45f3c8be-f10c-4194-8acf-4b12806660d1" (UID: "45f3c8be-f10c-4194-8acf-4b12806660d1"). InnerVolumeSpecName "kube-api-access-95n68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:07.090283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.090180 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca" pod "45f3c8be-f10c-4194-8acf-4b12806660d1" (UID: "45f3c8be-f10c-4194-8acf-4b12806660d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:26:07.189035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.188996 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.189035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.189027 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95n68\" (UniqueName: \"kubernetes.io/projected/45f3c8be-f10c-4194-8acf-4b12806660d1-kube-api-access-95n68\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.189035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.189038 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f3c8be-f10c-4194-8acf-4b12806660d1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:26:07.829827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.829796 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" Nov 03 01:26:07.830008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.829798 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk" event={"ID":"45f3c8be-f10c-4194-8acf-4b12806660d1","Type":"ContainerDied","Data":"ae38f86bbb9ec2499b4174f64b702a4a3ed2dd06d3fc883b2a96c0c9783bb07d"} Nov 03 01:26:07.830008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:07.829933 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae38f86bbb9ec2499b4174f64b702a4a3ed2dd06d3fc883b2a96c0c9783bb07d" Nov 03 01:26:12.154484 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:26:12.154445 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:26:14.379350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:14.379316 5144 scope.go:122] "RemoveContainer" containerID="9d89b1db16e525ec53b5e0bfe614270e8c0e46128949646dd06d732c75def8cf" Nov 03 01:26:14.391962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:14.391935 5144 scope.go:122] "RemoveContainer" containerID="7daedf6e6dfd7038d97a5f8b1533bca0de101f7dad7814c103df203154ef0b91" Nov 03 01:26:14.400714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:14.400693 5144 scope.go:122] "RemoveContainer" containerID="c7b71983069ea366dae3f610f4adab9da203b414e3b0a0777f3f500d526b4b8a" Nov 03 01:26:14.409104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:14.409058 5144 scope.go:122] "RemoveContainer" containerID="dceca7f152f8c15bd16dab37ff5b8b6f6b142d0ee5d77425468261544611c1a0" Nov 03 01:26:14.417182 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:14.417162 5144 scope.go:122] "RemoveContainer" containerID="c111d6c90605eb24edea7608e6774f390e0ab818e1146815ad51bb7a381f4e4a" Nov 03 01:26:14.425579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:14.425563 5144 scope.go:122] "RemoveContainer" containerID="3018253a96c52b601abaeac2bbecda9ea759284b8b81a8c7f27f7c346f3fb52f" Nov 03 01:26:15.491102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.491071 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n"] Nov 03 01:26:15.491564 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.491551 5144 cpu_manager.go:414] "containerMap: removing container" podUID="45f3c8be-f10c-4194-8acf-4b12806660d1" containerName="revoke-keys" Nov 03 01:26:15.491609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.491566 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f3c8be-f10c-4194-8acf-4b12806660d1" containerName="revoke-keys" Nov 03 01:26:15.491700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.491689 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f3c8be-f10c-4194-8acf-4b12806660d1" containerName="revoke-keys" Nov 03 01:26:15.496825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.496798 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.561866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.561824 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n"] Nov 03 01:26:15.596534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.596503 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:26:15.596733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.596581 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:26:15.678513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.678474 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.678728 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.678537 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.678787 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.678716 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmvvb\" (UniqueName: \"kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.779623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.779584 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.779853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.779660 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.779853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.779739 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmvvb\" (UniqueName: \"kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.780389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.780366 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.782185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.782154 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:15.836298 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:15.836265 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmvvb\" (UniqueName: \"kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb\") pod \"maas-api-revoke-sub-6b19338f-kkj9n\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:16.117846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:16.117764 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:16.493697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:16.493667 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n"] Nov 03 01:26:16.495391 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:26:16.495364 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eaf6efd_1b54_4c3b_9c13_92964de27870.slice/crio-8d33e6e8bdd1aa7898ecc54103f3ff22a85eabb10b7d09f8e2e19f868e44ed9a WatchSource:0}: Error finding container 8d33e6e8bdd1aa7898ecc54103f3ff22a85eabb10b7d09f8e2e19f868e44ed9a: Status 404 returned error can't find the container with id 8d33e6e8bdd1aa7898ecc54103f3ff22a85eabb10b7d09f8e2e19f868e44ed9a Nov 03 01:26:16.885011 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:16.884975 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" event={"ID":"9eaf6efd-1b54-4c3b-9c13-92964de27870","Type":"ContainerStarted","Data":"075b3e8d8f9f42a90e4125a035d4f85ee158c85230cacf031ef42bc426aa4c96"} Nov 03 01:26:16.885196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:16.885030 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" event={"ID":"9eaf6efd-1b54-4c3b-9c13-92964de27870","Type":"ContainerStarted","Data":"8d33e6e8bdd1aa7898ecc54103f3ff22a85eabb10b7d09f8e2e19f868e44ed9a"} Nov 03 01:26:16.999113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:16.999005 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" podStartSLOduration=1.998988647 podStartE2EDuration="1.998988647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:16.996836917 +0000 UTC m=+4867.142531393" watchObservedRunningTime="2026-11-03 01:26:16.998988647 +0000 UTC m=+4867.144683105" Nov 03 01:26:17.896551 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:17.896520 5144 generic.go:356] "Generic (PLEG): container finished" podID="9eaf6efd-1b54-4c3b-9c13-92964de27870" containerID="075b3e8d8f9f42a90e4125a035d4f85ee158c85230cacf031ef42bc426aa4c96" exitCode=0 Nov 03 01:26:17.896988 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:17.896604 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" event={"ID":"9eaf6efd-1b54-4c3b-9c13-92964de27870","Type":"ContainerDied","Data":"075b3e8d8f9f42a90e4125a035d4f85ee158c85230cacf031ef42bc426aa4c96"} Nov 03 01:26:18.060504 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:18.060472 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf"] Nov 03 01:26:18.062888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:18.062861 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f96db4-65dbf"] Nov 03 01:26:19.048051 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.048022 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:19.111954 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.111924 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb\" (UniqueName: \"kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb\") pod \"9eaf6efd-1b54-4c3b-9c13-92964de27870\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " Nov 03 01:26:19.112130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.111974 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca\") pod \"9eaf6efd-1b54-4c3b-9c13-92964de27870\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " Nov 03 01:26:19.112130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.111996 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca\") pod \"9eaf6efd-1b54-4c3b-9c13-92964de27870\" (UID: \"9eaf6efd-1b54-4c3b-9c13-92964de27870\") " Nov 03 01:26:19.112381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.112357 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca" pod "9eaf6efd-1b54-4c3b-9c13-92964de27870" (UID: "9eaf6efd-1b54-4c3b-9c13-92964de27870"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:26:19.114090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.114067 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca" pod "9eaf6efd-1b54-4c3b-9c13-92964de27870" (UID: "9eaf6efd-1b54-4c3b-9c13-92964de27870"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:26:19.114242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.114215 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb" pod "9eaf6efd-1b54-4c3b-9c13-92964de27870" (UID: "9eaf6efd-1b54-4c3b-9c13-92964de27870"). InnerVolumeSpecName "kube-api-access-tmvvb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:26:19.213163 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.213081 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmvvb\" (UniqueName: \"kubernetes.io/projected/9eaf6efd-1b54-4c3b-9c13-92964de27870-kube-api-access-tmvvb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:26:19.213163 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.213116 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:26:19.213163 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.213126 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9eaf6efd-1b54-4c3b-9c13-92964de27870-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:26:19.909742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.909708 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" Nov 03 01:26:19.909927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.909743 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n" event={"ID":"9eaf6efd-1b54-4c3b-9c13-92964de27870","Type":"ContainerDied","Data":"8d33e6e8bdd1aa7898ecc54103f3ff22a85eabb10b7d09f8e2e19f868e44ed9a"} Nov 03 01:26:19.909927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.909794 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d33e6e8bdd1aa7898ecc54103f3ff22a85eabb10b7d09f8e2e19f868e44ed9a" Nov 03 01:26:19.995511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:19.995477 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a3f4cae-8cdf-4626-bff4-b6af597531b4" path="/var/lib/kubelet/pods/9a3f4cae-8cdf-4626-bff4-b6af597531b4/volumes" Nov 03 01:26:31.348525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.348488 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4"] Nov 03 01:26:31.349029 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.349013 5144 cpu_manager.go:414] "containerMap: removing container" podUID="9eaf6efd-1b54-4c3b-9c13-92964de27870" containerName="revoke-keys" Nov 03 01:26:31.349083 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.349031 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9eaf6efd-1b54-4c3b-9c13-92964de27870" containerName="revoke-keys" Nov 03 01:26:31.349130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.349115 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="9eaf6efd-1b54-4c3b-9c13-92964de27870" containerName="revoke-keys" Nov 03 01:26:31.352323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.352308 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.410249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.410214 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4"] Nov 03 01:26:31.454099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.454075 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-a2af2b5f-kserve-self-signed-certs\"" Nov 03 01:26:31.457352 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.457327 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.457491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.457402 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.457491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.457450 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b55x5\" (UniqueName: \"kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.457651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.457516 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.457651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.457551 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.457651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.457603 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.558951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.558912 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.558983 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559005 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b55x5\" (UniqueName: \"kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559022 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559144 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559039 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559177 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559511 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559487 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559508 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.559624 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.559579 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.561437 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.561419 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.561701 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.561684 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.615915 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.615859 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b55x5\" (UniqueName: \"kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5\") pod \"e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:31.664622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:31.664594 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:32.039293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.039272 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4"] Nov 03 01:26:32.040220 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:26:32.040194 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91261a4c_c41c_4052_a896_8b263aa5fd23.slice/crio-3822195003ebbfbac432ae875973ba46d9019fe57e6a597abf77c3f0528643be WatchSource:0}: Error finding container 3822195003ebbfbac432ae875973ba46d9019fe57e6a597abf77c3f0528643be: Status 404 returned error can't find the container with id 3822195003ebbfbac432ae875973ba46d9019fe57e6a597abf77c3f0528643be Nov 03 01:26:32.068293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.068269 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf"] Nov 03 01:26:32.073246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.073230 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.124971 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.124947 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-a2af2b5f-epp-sa-dockercfg-2qc25\"" Nov 03 01:26:32.150188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.150158 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf"] Nov 03 01:26:32.166744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.166716 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjrjb\" (UniqueName: \"kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb\") pod \"e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.166886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.166766 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs\") pod \"e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.267927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.267893 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs\") pod \"e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.268113 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.268025 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjrjb\" (UniqueName: \"kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb\") pod \"e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.270707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.270681 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs\") pod \"e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.324529 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.324450 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjrjb\" (UniqueName: \"kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb\") pod \"e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.390626 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.390585 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:32.768208 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:32.768177 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf"] Nov 03 01:26:32.769279 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:26:32.769254 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd450ddc5_7e08_4c97_8ef3_33574d95e4ef.slice/crio-b1a1ca92a7b4c6cba5def0edaca026578c96346df3acb6188ebbcfb7d1c24a00 WatchSource:0}: Error finding container b1a1ca92a7b4c6cba5def0edaca026578c96346df3acb6188ebbcfb7d1c24a00: Status 404 returned error can't find the container with id b1a1ca92a7b4c6cba5def0edaca026578c96346df3acb6188ebbcfb7d1c24a00 Nov 03 01:26:33.002467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:33.002428 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" event={"ID":"91261a4c-c41c-4052-a896-8b263aa5fd23","Type":"ContainerStarted","Data":"0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac"} Nov 03 01:26:33.002699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:33.002476 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" event={"ID":"91261a4c-c41c-4052-a896-8b263aa5fd23","Type":"ContainerStarted","Data":"3822195003ebbfbac432ae875973ba46d9019fe57e6a597abf77c3f0528643be"} Nov 03 01:26:33.003978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:33.003945 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" event={"ID":"d450ddc5-7e08-4c97-8ef3-33574d95e4ef","Type":"ContainerStarted","Data":"b1a1ca92a7b4c6cba5def0edaca026578c96346df3acb6188ebbcfb7d1c24a00"} Nov 03 01:26:33.118370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:33.118233 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" podStartSLOduration=2.118217547 podStartE2EDuration="2.118217547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:33.115379362 +0000 UTC m=+4883.261073884" watchObservedRunningTime="2026-11-03 01:26:33.118217547 +0000 UTC m=+4883.263912005" Nov 03 01:26:33.665485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:33.665452 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:34.052134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:34.052102 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67"] Nov 03 01:26:34.099865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:34.099837 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0472eb8b-wfz67"] Nov 03 01:26:35.024082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:35.024044 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" event={"ID":"d450ddc5-7e08-4c97-8ef3-33574d95e4ef","Type":"ContainerStarted","Data":"17d5d6069a5493ef511c5f8ab5987ee3b9a9be126e31c3493c8710636f77704c"} Nov 03 01:26:35.024570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:35.024156 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:26:35.138126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:35.138073 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" podStartSLOduration=1.266053578 podStartE2EDuration="3.138055552s" totalImagesPullingTime="1.872001974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:35.135596494 +0000 UTC m=+4885.281290949" watchObservedRunningTime="2026-11-03 01:26:35.138055552 +0000 UTC m=+4885.283750005" Nov 03 01:26:35.997612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:35.997572 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c" path="/var/lib/kubelet/pods/ecbe4fb3-bff4-4f85-9a2e-8fa5c0a8db1c/volumes" Nov 03 01:26:41.665671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:41.665604 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:41.678286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:41.678260 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:42.109085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:26:42.109053 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:26:42.154655 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:26:42.154590 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:27:06.033291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:06.033266 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:27:07.044038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:07.044001 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg"] Nov 03 01:27:07.046453 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:07.046432 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35e71f1d-v5ldg"] Nov 03 01:27:07.995325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:07.995288 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581e7c72-05ed-4081-ba5c-76e46ae82e08" path="/var/lib/kubelet/pods/581e7c72-05ed-4081-ba5c-76e46ae82e08/volumes" Nov 03 01:27:12.155422 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:27:12.155377 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:27:14.470374 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:14.470339 5144 scope.go:122] "RemoveContainer" containerID="4a27b7ab35755a472225333adb6d7753f8278559af85ce1189edb43852932e94" Nov 03 01:27:24.047241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:24.047201 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt"] Nov 03 01:27:24.096372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:24.096345 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea3c67db-drmrt"] Nov 03 01:27:25.995814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:25.995779 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e759c769-df16-4098-9705-74507aaddf9b" path="/var/lib/kubelet/pods/e759c769-df16-4098-9705-74507aaddf9b/volumes" Nov 03 01:27:38.478907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:38.478864 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4"] Nov 03 01:27:38.483830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:38.483801 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf"] Nov 03 01:27:42.154726 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:27:42.154687 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:27:46.047772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:46.047724 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:27:53.502839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.502793 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" podUID="91261a4c-c41c-4052-a896-8b263aa5fd23" containerName="main" containerID="cri-o://0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac" gracePeriod=45 Nov 03 01:27:53.516736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.516681 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" containerID="cri-o://17d5d6069a5493ef511c5f8ab5987ee3b9a9be126e31c3493c8710636f77704c" gracePeriod=45 Nov 03 01:27:53.784033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.784005 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:27:53.874299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874258 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache\") pod \"91261a4c-c41c-4052-a896-8b263aa5fd23\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " Nov 03 01:27:53.874525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874379 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5\" (UniqueName: \"kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5\") pod \"91261a4c-c41c-4052-a896-8b263aa5fd23\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " Nov 03 01:27:53.874525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874429 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir\") pod \"91261a4c-c41c-4052-a896-8b263aa5fd23\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " Nov 03 01:27:53.874525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874462 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm\") pod \"91261a4c-c41c-4052-a896-8b263aa5fd23\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " Nov 03 01:27:53.874525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874496 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home\") pod \"91261a4c-c41c-4052-a896-8b263aa5fd23\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " Nov 03 01:27:53.874804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874534 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs\" (UniqueName: \"kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs\") pod \"91261a4c-c41c-4052-a896-8b263aa5fd23\" (UID: \"91261a4c-c41c-4052-a896-8b263aa5fd23\") " Nov 03 01:27:53.874804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874542 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache" pod "91261a4c-c41c-4052-a896-8b263aa5fd23" (UID: "91261a4c-c41c-4052-a896-8b263aa5fd23"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:53.874804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874770 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir" pod "91261a4c-c41c-4052-a896-8b263aa5fd23" (UID: "91261a4c-c41c-4052-a896-8b263aa5fd23"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:53.874965 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874815 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home" pod "91261a4c-c41c-4052-a896-8b263aa5fd23" (UID: "91261a4c-c41c-4052-a896-8b263aa5fd23"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:53.875028 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874963 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:53.875028 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.874986 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:53.875028 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.875001 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:53.876907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.876881 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm" pod "91261a4c-c41c-4052-a896-8b263aa5fd23" (UID: "91261a4c-c41c-4052-a896-8b263aa5fd23"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:27:53.876907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.876902 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5" pod "91261a4c-c41c-4052-a896-8b263aa5fd23" (UID: "91261a4c-c41c-4052-a896-8b263aa5fd23"). InnerVolumeSpecName "kube-api-access-b55x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:27:53.877098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.876920 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs" pod "91261a4c-c41c-4052-a896-8b263aa5fd23" (UID: "91261a4c-c41c-4052-a896-8b263aa5fd23"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:27:53.976426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.976384 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91261a4c-c41c-4052-a896-8b263aa5fd23-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:53.976426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.976421 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b55x5\" (UniqueName: \"kubernetes.io/projected/91261a4c-c41c-4052-a896-8b263aa5fd23-kube-api-access-b55x5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:53.976426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:53.976432 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91261a4c-c41c-4052-a896-8b263aa5fd23-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:27:54.539779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.539743 5144 generic.go:356] "Generic (PLEG): container finished" podID="91261a4c-c41c-4052-a896-8b263aa5fd23" containerID="0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac" exitCode=0 Nov 03 01:27:54.539779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.539783 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" Nov 03 01:27:54.540334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.539827 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" event={"ID":"91261a4c-c41c-4052-a896-8b263aa5fd23","Type":"ContainerDied","Data":"0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac"} Nov 03 01:27:54.540334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.539868 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4" event={"ID":"91261a4c-c41c-4052-a896-8b263aa5fd23","Type":"ContainerDied","Data":"3822195003ebbfbac432ae875973ba46d9019fe57e6a597abf77c3f0528643be"} Nov 03 01:27:54.540334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.539884 5144 scope.go:122] "RemoveContainer" containerID="0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac" Nov 03 01:27:54.549912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.549892 5144 scope.go:122] "RemoveContainer" containerID="0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac" Nov 03 01:27:54.550193 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:27:54.550175 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac\": container with ID starting with 0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac not found: ID does not exist" containerID="0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac" Nov 03 01:27:54.550276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.550201 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac"} err="failed to get container status \"0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac\": rpc error: code = NotFound desc = could not find container \"0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac\": container with ID starting with 0c2694ba1eb1eef14bfbfb92b659e56c0ea4aba9892f0aab951b560bab94beac not found: ID does not exist" Nov 03 01:27:54.703023 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.702991 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4"] Nov 03 01:27:54.752073 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:54.752042 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-5f8fb78bcb-5vtx4"] Nov 03 01:27:55.995816 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:55.995782 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91261a4c-c41c-4052-a896-8b263aa5fd23" path="/var/lib/kubelet/pods/91261a4c-c41c-4052-a896-8b263aa5fd23/volumes" Nov 03 01:27:56.032919 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:27:56.032877 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:06.033281 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:06.033243 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:06.033793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:06.033346 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:28:12.154043 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:28:12.154009 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:28:14.494137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:14.494107 5144 scope.go:122] "RemoveContainer" containerID="1f5b3dd1488777a0af3cd548e1cd6dfdab3aa9c8e93577d221a103cd614c0e27" Nov 03 01:28:14.507026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:14.507004 5144 scope.go:122] "RemoveContainer" containerID="7a8548c2968ffb32b0aa492235c93ce7f6e4d833565b5d9573c464e029bd1a1f" Nov 03 01:28:14.515504 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:14.515486 5144 scope.go:122] "RemoveContainer" containerID="b2cfb1dba2e4b5bb504c4ea2b3e6939001c62cb332d76172911f38182ef8674f" Nov 03 01:28:16.044965 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:16.044921 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:28:23.750036 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.749996 5144 generic.go:356] "Generic (PLEG): container finished" podID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerID="17d5d6069a5493ef511c5f8ab5987ee3b9a9be126e31c3493c8710636f77704c" exitCode=0 Nov 03 01:28:23.750464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.750068 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" event={"ID":"d450ddc5-7e08-4c97-8ef3-33574d95e4ef","Type":"ContainerDied","Data":"17d5d6069a5493ef511c5f8ab5987ee3b9a9be126e31c3493c8710636f77704c"} Nov 03 01:28:23.797038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.797015 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:28:23.906316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.906273 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb\" (UniqueName: \"kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb\") pod \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " Nov 03 01:28:23.906534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.906402 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs\") pod \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\" (UID: \"d450ddc5-7e08-4c97-8ef3-33574d95e4ef\") " Nov 03 01:28:23.908681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.908649 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs" pod "d450ddc5-7e08-4c97-8ef3-33574d95e4ef" (UID: "d450ddc5-7e08-4c97-8ef3-33574d95e4ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:28:23.908800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:23.908651 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb" pod "d450ddc5-7e08-4c97-8ef3-33574d95e4ef" (UID: "d450ddc5-7e08-4c97-8ef3-33574d95e4ef"). InnerVolumeSpecName "kube-api-access-gjrjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:28:24.008168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.008144 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjrjb\" (UniqueName: \"kubernetes.io/projected/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-kube-api-access-gjrjb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:28:24.008287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.008169 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d450ddc5-7e08-4c97-8ef3-33574d95e4ef-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:28:24.758366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.758332 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" Nov 03 01:28:24.758890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.758331 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf" event={"ID":"d450ddc5-7e08-4c97-8ef3-33574d95e4ef","Type":"ContainerDied","Data":"b1a1ca92a7b4c6cba5def0edaca026578c96346df3acb6188ebbcfb7d1c24a00"} Nov 03 01:28:24.758890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.758471 5144 scope.go:122] "RemoveContainer" containerID="17d5d6069a5493ef511c5f8ab5987ee3b9a9be126e31c3493c8710636f77704c" Nov 03 01:28:24.920991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.920959 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf"] Nov 03 01:28:24.929570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:24.929546 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-a2af2b5f-kserve-router-scheduler-5b67bdfc55-t6mwf"] Nov 03 01:28:25.995397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:25.995363 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" path="/var/lib/kubelet/pods/d450ddc5-7e08-4c97-8ef3-33574d95e4ef/volumes" Nov 03 01:28:29.296642 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.296592 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm"] Nov 03 01:28:29.299041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.297039 5144 cpu_manager.go:414] "containerMap: removing container" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" Nov 03 01:28:29.299041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.297050 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" Nov 03 01:28:29.299041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.297082 5144 cpu_manager.go:414] "containerMap: removing container" podUID="91261a4c-c41c-4052-a896-8b263aa5fd23" containerName="main" Nov 03 01:28:29.299041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.297087 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91261a4c-c41c-4052-a896-8b263aa5fd23" containerName="main" Nov 03 01:28:29.299041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.297168 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="91261a4c-c41c-4052-a896-8b263aa5fd23" containerName="main" Nov 03 01:28:29.299041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.297179 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="d450ddc5-7e08-4c97-8ef3-33574d95e4ef" containerName="main" Nov 03 01:28:29.300263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.300244 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.352390 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.352353 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6c9df1\"/\"openshift-service-ca.crt\"" Nov 03 01:28:29.361870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.361839 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.362033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.361887 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.362106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.362017 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.362174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.362123 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.362231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.362217 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.362303 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.362284 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sj4b\" (UniqueName: \"kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.401838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.401805 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-6c9df1\"/\"kube-root-ca.crt\"" Nov 03 01:28:29.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.401983 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6c9df1\"/\"default-dockercfg-rgj58\"" Nov 03 01:28:29.402017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.402001 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-6c9df1\"/\"simulator-single-main-6c9df1-kserve-self-signed-certs\"" Nov 03 01:28:29.404724 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.404702 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm"] Nov 03 01:28:29.463719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.463679 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.463903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.463750 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.463903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.463830 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.463903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.463885 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sj4b\" (UniqueName: \"kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.464043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.463941 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.464043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.463975 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.464137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.464115 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.464200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.464177 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.464270 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.464255 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.466294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.466273 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.466469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.466454 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.519938 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.519901 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sj4b\" (UniqueName: \"kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b\") pod \"simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.612176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.612086 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:29.993450 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:28:29.993415 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65b0e238_30ac_46bb_ac9c_c33878c9f774.slice/crio-57b7c77d6e56d876d1a054b6cd63a7d30573e2ca53d08e902583d1807e84e790 WatchSource:0}: Error finding container 57b7c77d6e56d876d1a054b6cd63a7d30573e2ca53d08e902583d1807e84e790: Status 404 returned error can't find the container with id 57b7c77d6e56d876d1a054b6cd63a7d30573e2ca53d08e902583d1807e84e790 Nov 03 01:28:29.996490 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:29.996468 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm"] Nov 03 01:28:30.797184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:30.797147 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" event={"ID":"65b0e238-30ac-46bb-ac9c-c33878c9f774","Type":"ContainerStarted","Data":"b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2"} Nov 03 01:28:30.797718 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:30.797200 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" event={"ID":"65b0e238-30ac-46bb-ac9c-c33878c9f774","Type":"ContainerStarted","Data":"57b7c77d6e56d876d1a054b6cd63a7d30573e2ca53d08e902583d1807e84e790"} Nov 03 01:28:30.911992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:30.911940 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" podStartSLOduration=1.9119261829999998 podStartE2EDuration="1.911926183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:28:30.910626118 +0000 UTC m=+5001.056320575" watchObservedRunningTime="2026-11-03 01:28:30.911926183 +0000 UTC m=+5001.057620668" Nov 03 01:28:31.612973 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:31.612945 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:39.612971 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:39.612934 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:39.625897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:39.625870 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:39.862050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:28:39.862018 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:28:42.155167 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:28:42.155128 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:29:12.154490 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:29:12.154456 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:29:14.549786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.549756 5144 scope.go:122] "RemoveContainer" containerID="a56cd0c7d684b6ed1e02b3e374adadbc2196d2d9d8a167876f8e3f329c726d72" Nov 03 01:29:14.778279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.778235 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx"] Nov 03 01:29:14.783602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.783576 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:14.838353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.838257 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx"] Nov 03 01:29:14.884563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.884523 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:29:14.884755 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.884580 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:29:14.955034 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.954995 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gclkh\" (UniqueName: \"kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:14.955034 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.955036 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:14.955333 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:14.955060 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.056403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.056364 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gclkh\" (UniqueName: \"kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.056614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.056412 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.056614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.056442 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.057190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.057157 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.058876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.058854 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.113124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.113050 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gclkh\" (UniqueName: \"kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh\") pod \"maas-api-revoke-sub-4b1ce5a3-r5tpx\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.396661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.396543 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:15.769167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:15.769136 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx"] Nov 03 01:29:15.770811 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:29:15.770781 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110558df_044b_435c_b088_ce718e0cf8be.slice/crio-5c3b6b6433c48480380fc0db9d4c096020fbc864d1e3ee1d3b66fdb2c711c4f1 WatchSource:0}: Error finding container 5c3b6b6433c48480380fc0db9d4c096020fbc864d1e3ee1d3b66fdb2c711c4f1: Status 404 returned error can't find the container with id 5c3b6b6433c48480380fc0db9d4c096020fbc864d1e3ee1d3b66fdb2c711c4f1 Nov 03 01:29:16.089231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:16.089190 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" event={"ID":"110558df-044b-435c-b088-ce718e0cf8be","Type":"ContainerStarted","Data":"52caf10591cf88aaf9bad8d39e3f2c47274d7a335d83063e85697fbf7d727bf4"} Nov 03 01:29:16.089408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:16.089252 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" event={"ID":"110558df-044b-435c-b088-ce718e0cf8be","Type":"ContainerStarted","Data":"5c3b6b6433c48480380fc0db9d4c096020fbc864d1e3ee1d3b66fdb2c711c4f1"} Nov 03 01:29:16.203268 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:16.203210 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" podStartSLOduration=2.203197129 podStartE2EDuration="2.203197129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:29:16.200658932 +0000 UTC m=+5046.346353410" watchObservedRunningTime="2026-11-03 01:29:16.203197129 +0000 UTC m=+5046.348891585" Nov 03 01:29:17.097410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:17.097377 5144 generic.go:356] "Generic (PLEG): container finished" podID="110558df-044b-435c-b088-ce718e0cf8be" containerID="52caf10591cf88aaf9bad8d39e3f2c47274d7a335d83063e85697fbf7d727bf4" exitCode=0 Nov 03 01:29:17.097876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:17.097469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" event={"ID":"110558df-044b-435c-b088-ce718e0cf8be","Type":"ContainerDied","Data":"52caf10591cf88aaf9bad8d39e3f2c47274d7a335d83063e85697fbf7d727bf4"} Nov 03 01:29:18.250388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.250358 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:18.289236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.289203 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca\") pod \"110558df-044b-435c-b088-ce718e0cf8be\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " Nov 03 01:29:18.289400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.289269 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca\") pod \"110558df-044b-435c-b088-ce718e0cf8be\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " Nov 03 01:29:18.289457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.289396 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh\" (UniqueName: \"kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh\") pod \"110558df-044b-435c-b088-ce718e0cf8be\" (UID: \"110558df-044b-435c-b088-ce718e0cf8be\") " Nov 03 01:29:18.289648 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.289611 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca" pod "110558df-044b-435c-b088-ce718e0cf8be" (UID: "110558df-044b-435c-b088-ce718e0cf8be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:29:18.289827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.289803 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/110558df-044b-435c-b088-ce718e0cf8be-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:18.291671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.291621 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca" pod "110558df-044b-435c-b088-ce718e0cf8be" (UID: "110558df-044b-435c-b088-ce718e0cf8be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:29:18.291793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.291721 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh" pod "110558df-044b-435c-b088-ce718e0cf8be" (UID: "110558df-044b-435c-b088-ce718e0cf8be"). InnerVolumeSpecName "kube-api-access-gclkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:18.391290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.391214 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gclkh\" (UniqueName: \"kubernetes.io/projected/110558df-044b-435c-b088-ce718e0cf8be-kube-api-access-gclkh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:18.391290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:18.391244 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/110558df-044b-435c-b088-ce718e0cf8be-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:19.111622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:19.111588 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" Nov 03 01:29:19.111846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:19.111620 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx" event={"ID":"110558df-044b-435c-b088-ce718e0cf8be","Type":"ContainerDied","Data":"5c3b6b6433c48480380fc0db9d4c096020fbc864d1e3ee1d3b66fdb2c711c4f1"} Nov 03 01:29:19.111846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:19.111703 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c3b6b6433c48480380fc0db9d4c096020fbc864d1e3ee1d3b66fdb2c711c4f1" Nov 03 01:29:27.045232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:27.045202 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp"] Nov 03 01:29:27.094430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:27.094395 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hq8xp"] Nov 03 01:29:27.995249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:27.995214 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32fe01be-f261-4580-a8aa-475d700433b5" path="/var/lib/kubelet/pods/32fe01be-f261-4580-a8aa-475d700433b5/volumes" Nov 03 01:29:30.865006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:30.864971 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm"] Nov 03 01:29:42.154398 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:29:42.154357 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:29:45.885392 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:45.885339 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" podUID="65b0e238-30ac-46bb-ac9c-c33878c9f774" containerName="main" containerID="cri-o://b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2" gracePeriod=45 Nov 03 01:29:46.136006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.135943 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:29:46.251794 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.251768 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm\") pod \"65b0e238-30ac-46bb-ac9c-c33878c9f774\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " Nov 03 01:29:46.251949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.251829 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home\") pod \"65b0e238-30ac-46bb-ac9c-c33878c9f774\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " Nov 03 01:29:46.251949 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.251909 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b\" (UniqueName: \"kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b\") pod \"65b0e238-30ac-46bb-ac9c-c33878c9f774\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " Nov 03 01:29:46.252085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.251952 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache\") pod \"65b0e238-30ac-46bb-ac9c-c33878c9f774\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " Nov 03 01:29:46.252085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.251973 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir\") pod \"65b0e238-30ac-46bb-ac9c-c33878c9f774\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " Nov 03 01:29:46.252085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.251996 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs\") pod \"65b0e238-30ac-46bb-ac9c-c33878c9f774\" (UID: \"65b0e238-30ac-46bb-ac9c-c33878c9f774\") " Nov 03 01:29:46.252258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.252143 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home" pod "65b0e238-30ac-46bb-ac9c-c33878c9f774" (UID: "65b0e238-30ac-46bb-ac9c-c33878c9f774"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:46.252258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.252192 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache" pod "65b0e238-30ac-46bb-ac9c-c33878c9f774" (UID: "65b0e238-30ac-46bb-ac9c-c33878c9f774"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:46.252382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.252256 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir" pod "65b0e238-30ac-46bb-ac9c-c33878c9f774" (UID: "65b0e238-30ac-46bb-ac9c-c33878c9f774"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:46.252382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.252340 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:46.252382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.252356 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:46.252382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.252367 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:46.254173 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.254149 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm" pod "65b0e238-30ac-46bb-ac9c-c33878c9f774" (UID: "65b0e238-30ac-46bb-ac9c-c33878c9f774"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:29:46.254297 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.254286 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs" pod "65b0e238-30ac-46bb-ac9c-c33878c9f774" (UID: "65b0e238-30ac-46bb-ac9c-c33878c9f774"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:29:46.254450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.254430 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b" pod "65b0e238-30ac-46bb-ac9c-c33878c9f774" (UID: "65b0e238-30ac-46bb-ac9c-c33878c9f774"). InnerVolumeSpecName "kube-api-access-5sj4b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:29:46.269647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.269599 5144 generic.go:356] "Generic (PLEG): container finished" podID="65b0e238-30ac-46bb-ac9c-c33878c9f774" containerID="b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2" exitCode=0 Nov 03 01:29:46.269753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.269709 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" Nov 03 01:29:46.269833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.269750 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" event={"ID":"65b0e238-30ac-46bb-ac9c-c33878c9f774","Type":"ContainerDied","Data":"b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2"} Nov 03 01:29:46.269833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.269796 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm" event={"ID":"65b0e238-30ac-46bb-ac9c-c33878c9f774","Type":"ContainerDied","Data":"57b7c77d6e56d876d1a054b6cd63a7d30573e2ca53d08e902583d1807e84e790"} Nov 03 01:29:46.269833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.269822 5144 scope.go:122] "RemoveContainer" containerID="b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2" Nov 03 01:29:46.284257 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.284238 5144 scope.go:122] "RemoveContainer" containerID="b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2" Nov 03 01:29:46.284582 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:29:46.284562 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2\": container with ID starting with b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2 not found: ID does not exist" containerID="b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2" Nov 03 01:29:46.284753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.284586 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2"} err="failed to get container status \"b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2\": rpc error: code = NotFound desc = could not find container \"b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2\": container with ID starting with b98cf99c50509176373ab1d0ab2a90e77ff7268be6786b0e1f49278094dd1ba2 not found: ID does not exist" Nov 03 01:29:46.352909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.352881 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65b0e238-30ac-46bb-ac9c-c33878c9f774-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:46.352909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.352905 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65b0e238-30ac-46bb-ac9c-c33878c9f774-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:46.353093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.352916 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sj4b\" (UniqueName: \"kubernetes.io/projected/65b0e238-30ac-46bb-ac9c-c33878c9f774-kube-api-access-5sj4b\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:29:46.431844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.431773 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm"] Nov 03 01:29:46.436775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:46.436753 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-6c9df1/simulator-single-main-6c9df1-kserve-55997b86fc-7zgtm"] Nov 03 01:29:47.995013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:29:47.994978 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65b0e238-30ac-46bb-ac9c-c33878c9f774" path="/var/lib/kubelet/pods/65b0e238-30ac-46bb-ac9c-c33878c9f774/volumes" Nov 03 01:30:00.158582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.158548 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w"] Nov 03 01:30:00.159058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.158971 5144 cpu_manager.go:414] "containerMap: removing container" podUID="65b0e238-30ac-46bb-ac9c-c33878c9f774" containerName="main" Nov 03 01:30:00.159058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.158983 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65b0e238-30ac-46bb-ac9c-c33878c9f774" containerName="main" Nov 03 01:30:00.159058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.159006 5144 cpu_manager.go:414] "containerMap: removing container" podUID="110558df-044b-435c-b088-ce718e0cf8be" containerName="revoke-keys" Nov 03 01:30:00.159058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.159010 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="110558df-044b-435c-b088-ce718e0cf8be" containerName="revoke-keys" Nov 03 01:30:00.159197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.159087 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="110558df-044b-435c-b088-ce718e0cf8be" containerName="revoke-keys" Nov 03 01:30:00.159197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.159101 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="65b0e238-30ac-46bb-ac9c-c33878c9f774" containerName="main" Nov 03 01:30:00.162116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.162101 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.217859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.217834 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w"] Nov 03 01:30:00.263813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.263792 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:30:00.269133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.269109 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfg5f\" (UniqueName: \"kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f\") pod \"maas-api-key-cleanup-29894490-srh8w\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.269243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.269230 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-srh8w\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.370692 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.370660 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-srh8w\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.370841 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.370751 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfg5f\" (UniqueName: \"kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f\") pod \"maas-api-key-cleanup-29894490-srh8w\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.371251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.371230 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894490-srh8w\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.427126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.427066 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfg5f\" (UniqueName: \"kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f\") pod \"maas-api-key-cleanup-29894490-srh8w\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.474702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.474674 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:00.842936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:00.842909 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w"] Nov 03 01:30:00.843878 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:30:00.843847 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9256470_eefd_4237_83b2_2aa2a9890bbe.slice/crio-99516bc3a0d9447fd4cae4adcd44427c1e573c723e3395cd956575c7977ed81c WatchSource:0}: Error finding container 99516bc3a0d9447fd4cae4adcd44427c1e573c723e3395cd956575c7977ed81c: Status 404 returned error can't find the container with id 99516bc3a0d9447fd4cae4adcd44427c1e573c723e3395cd956575c7977ed81c Nov 03 01:30:01.355930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:01.355895 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerStarted","Data":"4c3d10973c6674c59986db57becada606748e2558fc4940c4e10d052af5f56e6"} Nov 03 01:30:01.356320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:01.355945 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerStarted","Data":"99516bc3a0d9447fd4cae4adcd44427c1e573c723e3395cd956575c7977ed81c"} Nov 03 01:30:01.468434 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:01.468335 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" podStartSLOduration=1.468320578 podStartE2EDuration="1.468320578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:01.466691942 +0000 UTC m=+5091.612386416" watchObservedRunningTime="2026-11-03 01:30:01.468320578 +0000 UTC m=+5091.614015101" Nov 03 01:30:02.362600 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:02.362551 5144 generic.go:356] "Generic (PLEG): container finished" podID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerID="4c3d10973c6674c59986db57becada606748e2558fc4940c4e10d052af5f56e6" exitCode=60 Nov 03 01:30:02.363026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:02.362666 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerDied","Data":"4c3d10973c6674c59986db57becada606748e2558fc4940c4e10d052af5f56e6"} Nov 03 01:30:02.363026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:02.362868 5144 scope.go:122] "RemoveContainer" containerID="4c3d10973c6674c59986db57becada606748e2558fc4940c4e10d052af5f56e6" Nov 03 01:30:03.374179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:03.374102 5144 generic.go:356] "Generic (PLEG): container finished" podID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerID="436a4ec1b8420241d4615ab27b20877426d5e4d5612a07a1bbcb0e8603b6544e" exitCode=60 Nov 03 01:30:03.374179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:03.374152 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerDied","Data":"436a4ec1b8420241d4615ab27b20877426d5e4d5612a07a1bbcb0e8603b6544e"} Nov 03 01:30:03.374759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:03.374223 5144 scope.go:122] "RemoveContainer" containerID="4c3d10973c6674c59986db57becada606748e2558fc4940c4e10d052af5f56e6" Nov 03 01:30:03.374759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:03.374448 5144 scope.go:122] "RemoveContainer" containerID="436a4ec1b8420241d4615ab27b20877426d5e4d5612a07a1bbcb0e8603b6544e" Nov 03 01:30:03.374759 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:30:03.374724 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-srh8w_odh-ai-gateway-infra(e9256470-eefd-4237-83b2-2aa2a9890bbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" Nov 03 01:30:04.381797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:04.381768 5144 scope.go:122] "RemoveContainer" containerID="436a4ec1b8420241d4615ab27b20877426d5e4d5612a07a1bbcb0e8603b6544e" Nov 03 01:30:04.382170 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:30:04.381942 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-srh8w_odh-ai-gateway-infra(e9256470-eefd-4237-83b2-2aa2a9890bbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" Nov 03 01:30:11.209098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:11.209062 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:30:11.219775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:11.219755 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:30:12.043852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:12.043819 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8"] Nov 03 01:30:12.093303 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:12.093275 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f904bff9-47wp8"] Nov 03 01:30:12.157587 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:30:12.157550 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:30:13.995247 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:13.995215 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="122c7943-a9e7-4bf0-b84f-bd3d1fd0d247" path="/var/lib/kubelet/pods/122c7943-a9e7-4bf0-b84f-bd3d1fd0d247/volumes" Nov 03 01:30:14.585732 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:14.585703 5144 scope.go:122] "RemoveContainer" containerID="bf4e5a2ed73e050f7f9510f708bc2a3fd7b190b1114c2440ee06bdfd11c2bfeb" Nov 03 01:30:14.988566 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:14.988484 5144 scope.go:122] "RemoveContainer" containerID="436a4ec1b8420241d4615ab27b20877426d5e4d5612a07a1bbcb0e8603b6544e" Nov 03 01:30:15.447557 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:15.447521 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerStarted","Data":"de22dc3a1e3f2aa979ad445e091888fb600e216b40bf3ea3f7e822636fd4b789"} Nov 03 01:30:16.454727 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:16.454693 5144 generic.go:356] "Generic (PLEG): container finished" podID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerID="de22dc3a1e3f2aa979ad445e091888fb600e216b40bf3ea3f7e822636fd4b789" exitCode=60 Nov 03 01:30:16.455210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:16.454753 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerDied","Data":"de22dc3a1e3f2aa979ad445e091888fb600e216b40bf3ea3f7e822636fd4b789"} Nov 03 01:30:16.455210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:16.454811 5144 scope.go:122] "RemoveContainer" containerID="436a4ec1b8420241d4615ab27b20877426d5e4d5612a07a1bbcb0e8603b6544e" Nov 03 01:30:16.455210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:16.455131 5144 scope.go:122] "RemoveContainer" containerID="de22dc3a1e3f2aa979ad445e091888fb600e216b40bf3ea3f7e822636fd4b789" Nov 03 01:30:16.455370 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:30:16.455346 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894490-srh8w_odh-ai-gateway-infra(e9256470-eefd-4237-83b2-2aa2a9890bbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" Nov 03 01:30:16.579267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:16.579237 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w"] Nov 03 01:30:17.601858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.601835 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:17.633162 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.633134 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca\") pod \"e9256470-eefd-4237-83b2-2aa2a9890bbe\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " Nov 03 01:30:17.633288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.633203 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f\" (UniqueName: \"kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f\") pod \"e9256470-eefd-4237-83b2-2aa2a9890bbe\" (UID: \"e9256470-eefd-4237-83b2-2aa2a9890bbe\") " Nov 03 01:30:17.633515 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.633491 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca" pod "e9256470-eefd-4237-83b2-2aa2a9890bbe" (UID: "e9256470-eefd-4237-83b2-2aa2a9890bbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:30:17.633586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.633553 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9256470-eefd-4237-83b2-2aa2a9890bbe-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:17.635375 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.635347 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f" pod "e9256470-eefd-4237-83b2-2aa2a9890bbe" (UID: "e9256470-eefd-4237-83b2-2aa2a9890bbe"). InnerVolumeSpecName "kube-api-access-kfg5f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:30:17.735262 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:17.735166 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfg5f\" (UniqueName: \"kubernetes.io/projected/e9256470-eefd-4237-83b2-2aa2a9890bbe-kube-api-access-kfg5f\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:30:18.468521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:18.468495 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" Nov 03 01:30:18.468719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:18.468532 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w" event={"ID":"e9256470-eefd-4237-83b2-2aa2a9890bbe","Type":"ContainerDied","Data":"99516bc3a0d9447fd4cae4adcd44427c1e573c723e3395cd956575c7977ed81c"} Nov 03 01:30:18.468719 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:18.468590 5144 scope.go:122] "RemoveContainer" containerID="de22dc3a1e3f2aa979ad445e091888fb600e216b40bf3ea3f7e822636fd4b789" Nov 03 01:30:18.630382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:18.630347 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w"] Nov 03 01:30:18.679492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:18.679463 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894490-srh8w"] Nov 03 01:30:19.994878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:19.994846 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" path="/var/lib/kubelet/pods/e9256470-eefd-4237-83b2-2aa2a9890bbe/volumes" Nov 03 01:30:27.258548 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.258515 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85"] Nov 03 01:30:27.258991 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.258976 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.259060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.258993 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.259060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.259003 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.259060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.259008 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.259171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.259098 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.259171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.259109 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.259171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.259115 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:30:27.262252 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.262234 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.322824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.322792 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.322998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.322850 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.322998 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.322935 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.323128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.322998 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.323128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.323034 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.323128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.323113 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tvpt\" (UniqueName: \"kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.339523 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.339492 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85"] Nov 03 01:30:27.364328 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.364300 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-7b3137d3-kserve-self-signed-certs\"" Nov 03 01:30:27.423948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.423771 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tvpt\" (UniqueName: \"kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.423948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.423948 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424011 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424041 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424120 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424153 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424433 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424516 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424486 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.424553 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.424509 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.426755 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.426732 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.426890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.426839 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.480702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.480673 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tvpt\" (UniqueName: \"kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt\") pod \"e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.573326 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.573253 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:27.959923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:27.959899 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85"] Nov 03 01:30:27.961563 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:30:27.961537 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ba736bd_c7fa_48e5_86f9_013aaea7beb7.slice/crio-4a6428da58280e61e70919493718ac41d3603805545cf049ded74f924ad67f3b WatchSource:0}: Error finding container 4a6428da58280e61e70919493718ac41d3603805545cf049ded74f924ad67f3b: Status 404 returned error can't find the container with id 4a6428da58280e61e70919493718ac41d3603805545cf049ded74f924ad67f3b Nov 03 01:30:28.526534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:28.526497 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" event={"ID":"5ba736bd-c7fa-48e5-86f9-013aaea7beb7","Type":"ContainerStarted","Data":"7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64"} Nov 03 01:30:28.527062 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:28.526546 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" event={"ID":"5ba736bd-c7fa-48e5-86f9-013aaea7beb7","Type":"ContainerStarted","Data":"4a6428da58280e61e70919493718ac41d3603805545cf049ded74f924ad67f3b"} Nov 03 01:30:28.573945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:28.573919 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:28.640483 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:28.640436 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" podStartSLOduration=1.6404235900000002 podStartE2EDuration="1.64042359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:30:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:30:28.639152983 +0000 UTC m=+5118.784847486" watchObservedRunningTime="2026-11-03 01:30:28.64042359 +0000 UTC m=+5118.786118075" Nov 03 01:30:37.573423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:37.573384 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:37.585838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:37.585806 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:37.621525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:37.621499 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:30:42.153048 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:30:42.153015 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:30:46.043123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:46.043088 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v"] Nov 03 01:30:46.092299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:46.092262 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-hjs8v"] Nov 03 01:30:47.994832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:30:47.994797 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bd80da7-eb07-45a5-bc94-b131abb474b1" path="/var/lib/kubelet/pods/1bd80da7-eb07-45a5-bc94-b131abb474b1/volumes" Nov 03 01:31:07.051989 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:07.051953 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk"] Nov 03 01:31:07.101244 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:07.101216 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-7w2qk"] Nov 03 01:31:07.994970 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:07.994935 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f3c8be-f10c-4194-8acf-4b12806660d1" path="/var/lib/kubelet/pods/45f3c8be-f10c-4194-8acf-4b12806660d1/volumes" Nov 03 01:31:12.154744 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:31:12.154699 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:31:14.613429 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:14.613398 5144 scope.go:122] "RemoveContainer" containerID="723a6992a56d7e60821462dbce2d59b2d19b4b94c8197a68c36cb0f2079c076a" Nov 03 01:31:14.623056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:14.623030 5144 scope.go:122] "RemoveContainer" containerID="2fd42157ce1c2a8b8ecfa533855766bf6d16284249e95a53d51262567525b346" Nov 03 01:31:19.043414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:19.043332 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n"] Nov 03 01:31:19.092855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:19.092821 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-kkj9n"] Nov 03 01:31:19.995941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:19.995909 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9eaf6efd-1b54-4c3b-9c13-92964de27870" path="/var/lib/kubelet/pods/9eaf6efd-1b54-4c3b-9c13-92964de27870/volumes" Nov 03 01:31:24.231302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.231262 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf"] Nov 03 01:31:24.231975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.231954 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:31:24.232074 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.231977 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9256470-eefd-4237-83b2-2aa2a9890bbe" containerName="cleanup" Nov 03 01:31:24.235196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.235176 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.297930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.297894 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.297930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.297943 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghnsb\" (UniqueName: \"kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.298157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.298085 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.336804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.336771 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:31:24.337016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.336861 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:31:24.339288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.339263 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf"] Nov 03 01:31:24.398946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.398909 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.399134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.398954 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghnsb\" (UniqueName: \"kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.399134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.398986 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.399683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.399654 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.401446 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.401424 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.455419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.455378 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghnsb\" (UniqueName: \"kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb\") pod \"maas-api-revoke-sub-452b6d7b-bvfzf\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.548305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.548265 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:24.921026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.920997 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf"] Nov 03 01:31:24.923998 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:31:24.923968 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc5ff267_22df_467c_a2a4_068c10e9bc2e.slice/crio-f07b2540ba13a57236a077cd063e88e6664ddaf9de1bb4ecea108fc6bd9a2ee7 WatchSource:0}: Error finding container f07b2540ba13a57236a077cd063e88e6664ddaf9de1bb4ecea108fc6bd9a2ee7: Status 404 returned error can't find the container with id f07b2540ba13a57236a077cd063e88e6664ddaf9de1bb4ecea108fc6bd9a2ee7 Nov 03 01:31:24.937785 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:24.937747 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" event={"ID":"cc5ff267-22df-467c-a2a4-068c10e9bc2e","Type":"ContainerStarted","Data":"f07b2540ba13a57236a077cd063e88e6664ddaf9de1bb4ecea108fc6bd9a2ee7"} Nov 03 01:31:25.945364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:25.945320 5144 generic.go:356] "Generic (PLEG): container finished" podID="cc5ff267-22df-467c-a2a4-068c10e9bc2e" containerID="62448ddf8c272fb263b4093e5a542d20f9f74639a3a77b10368775e914651c91" exitCode=0 Nov 03 01:31:25.945876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:25.945405 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" event={"ID":"cc5ff267-22df-467c-a2a4-068c10e9bc2e","Type":"ContainerDied","Data":"62448ddf8c272fb263b4093e5a542d20f9f74639a3a77b10368775e914651c91"} Nov 03 01:31:27.180387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.180026 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:27.240726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.233110 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca\") pod \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " Nov 03 01:31:27.241310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.241165 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca" pod "cc5ff267-22df-467c-a2a4-068c10e9bc2e" (UID: "cc5ff267-22df-467c-a2a4-068c10e9bc2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:27.241522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.241500 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb\" (UniqueName: \"kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb\") pod \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " Nov 03 01:31:27.242182 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.242162 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca\") pod \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\" (UID: \"cc5ff267-22df-467c-a2a4-068c10e9bc2e\") " Nov 03 01:31:27.243216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.243187 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:27.257958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.257892 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb" pod "cc5ff267-22df-467c-a2a4-068c10e9bc2e" (UID: "cc5ff267-22df-467c-a2a4-068c10e9bc2e"). InnerVolumeSpecName "kube-api-access-ghnsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:27.258272 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.258248 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca" pod "cc5ff267-22df-467c-a2a4-068c10e9bc2e" (UID: "cc5ff267-22df-467c-a2a4-068c10e9bc2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:27.348659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.344346 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghnsb\" (UniqueName: \"kubernetes.io/projected/cc5ff267-22df-467c-a2a4-068c10e9bc2e-kube-api-access-ghnsb\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:27.348659 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.344392 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc5ff267-22df-467c-a2a4-068c10e9bc2e-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:27.987729 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.987693 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" Nov 03 01:31:27.988080 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.987867 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf" event={"ID":"cc5ff267-22df-467c-a2a4-068c10e9bc2e","Type":"ContainerDied","Data":"f07b2540ba13a57236a077cd063e88e6664ddaf9de1bb4ecea108fc6bd9a2ee7"} Nov 03 01:31:27.988333 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:27.988305 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f07b2540ba13a57236a077cd063e88e6664ddaf9de1bb4ecea108fc6bd9a2ee7" Nov 03 01:31:37.001978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:37.001943 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85"] Nov 03 01:31:42.155814 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:31:42.155771 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:31:49.701295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.701260 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh"] Nov 03 01:31:49.701884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.701795 5144 cpu_manager.go:414] "containerMap: removing container" podUID="cc5ff267-22df-467c-a2a4-068c10e9bc2e" containerName="revoke-keys" Nov 03 01:31:49.701884 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.701810 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc5ff267-22df-467c-a2a4-068c10e9bc2e" containerName="revoke-keys" Nov 03 01:31:49.702009 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.701929 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc5ff267-22df-467c-a2a4-068c10e9bc2e" containerName="revoke-keys" Nov 03 01:31:49.705340 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.705322 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.734608 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.734568 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.734852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.734619 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsjk7\" (UniqueName: \"kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.734852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.734746 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.760913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.760878 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:31:49.806901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.806862 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:31:49.810554 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.810513 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh"] Nov 03 01:31:49.835477 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.835433 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.835875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.835593 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.835875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.835672 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsjk7\" (UniqueName: \"kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.836338 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.836314 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.838414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.838373 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:49.892393 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:49.892355 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsjk7\" (UniqueName: \"kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7\") pod \"maas-api-revoke-sub-3e31a506-v5wlh\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:50.023424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:50.023383 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:50.399066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:50.399030 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh"] Nov 03 01:31:50.400347 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:31:50.400322 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc592391_fc11_4097_86b4_2b7e9290ccb7.slice/crio-ca5c0546e0017160db65332c867555ca751eb6639333b62841a1a3812f0812cc WatchSource:0}: Error finding container ca5c0546e0017160db65332c867555ca751eb6639333b62841a1a3812f0812cc: Status 404 returned error can't find the container with id ca5c0546e0017160db65332c867555ca751eb6639333b62841a1a3812f0812cc Nov 03 01:31:51.134066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:51.134029 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" event={"ID":"bc592391-fc11-4097-86b4-2b7e9290ccb7","Type":"ContainerStarted","Data":"aadf5c86e943eb9bc696fea4c8a079b4244767d2d8e88dcbcf43b593f59b2dc9"} Nov 03 01:31:51.134714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:51.134088 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" event={"ID":"bc592391-fc11-4097-86b4-2b7e9290ccb7","Type":"ContainerStarted","Data":"ca5c0546e0017160db65332c867555ca751eb6639333b62841a1a3812f0812cc"} Nov 03 01:31:51.247008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:51.246944 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" podStartSLOduration=2.246928305 podStartE2EDuration="2.246928305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:31:51.245583505 +0000 UTC m=+5201.391277988" watchObservedRunningTime="2026-11-03 01:31:51.246928305 +0000 UTC m=+5201.392622752" Nov 03 01:31:52.029188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.029117 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" podUID="5ba736bd-c7fa-48e5-86f9-013aaea7beb7" containerName="main" containerID="cri-o://7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64" gracePeriod=45 Nov 03 01:31:52.146224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.146183 5144 generic.go:356] "Generic (PLEG): container finished" podID="bc592391-fc11-4097-86b4-2b7e9290ccb7" containerID="aadf5c86e943eb9bc696fea4c8a079b4244767d2d8e88dcbcf43b593f59b2dc9" exitCode=0 Nov 03 01:31:52.146798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.146387 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" event={"ID":"bc592391-fc11-4097-86b4-2b7e9290ccb7","Type":"ContainerDied","Data":"aadf5c86e943eb9bc696fea4c8a079b4244767d2d8e88dcbcf43b593f59b2dc9"} Nov 03 01:31:52.339804 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.339771 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:31:52.366089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366039 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt\" (UniqueName: \"kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt\") pod \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " Nov 03 01:31:52.366314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366139 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir\") pod \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " Nov 03 01:31:52.366314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366172 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache\") pod \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " Nov 03 01:31:52.366314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366239 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm\") pod \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " Nov 03 01:31:52.366314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366284 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs\") pod \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " Nov 03 01:31:52.366563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366349 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home\") pod \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\" (UID: \"5ba736bd-c7fa-48e5-86f9-013aaea7beb7\") " Nov 03 01:31:52.367343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.366988 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home" pod "5ba736bd-c7fa-48e5-86f9-013aaea7beb7" (UID: "5ba736bd-c7fa-48e5-86f9-013aaea7beb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:52.367606 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.367464 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache" pod "5ba736bd-c7fa-48e5-86f9-013aaea7beb7" (UID: "5ba736bd-c7fa-48e5-86f9-013aaea7beb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:52.367764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.367712 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir" pod "5ba736bd-c7fa-48e5-86f9-013aaea7beb7" (UID: "5ba736bd-c7fa-48e5-86f9-013aaea7beb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:52.369983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.369785 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt" pod "5ba736bd-c7fa-48e5-86f9-013aaea7beb7" (UID: "5ba736bd-c7fa-48e5-86f9-013aaea7beb7"). InnerVolumeSpecName "kube-api-access-2tvpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:52.370335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.370308 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm" pod "5ba736bd-c7fa-48e5-86f9-013aaea7beb7" (UID: "5ba736bd-c7fa-48e5-86f9-013aaea7beb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:31:52.372579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.372551 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs" pod "5ba736bd-c7fa-48e5-86f9-013aaea7beb7" (UID: "5ba736bd-c7fa-48e5-86f9-013aaea7beb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:52.467821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.467685 5144 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-home\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:52.467821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.467733 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tvpt\" (UniqueName: \"kubernetes.io/projected/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-kube-api-access-2tvpt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:52.467821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.467750 5144 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tmp-dir\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:52.467821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.467765 5144 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-model-cache\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:52.467821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.467779 5144 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-dshm\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:52.467821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:52.467793 5144 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ba736bd-c7fa-48e5-86f9-013aaea7beb7-tls-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:53.177656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.177461 5144 generic.go:356] "Generic (PLEG): container finished" podID="5ba736bd-c7fa-48e5-86f9-013aaea7beb7" containerID="7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64" exitCode=0 Nov 03 01:31:53.177656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.177595 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" Nov 03 01:31:53.178280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.177684 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" event={"ID":"5ba736bd-c7fa-48e5-86f9-013aaea7beb7","Type":"ContainerDied","Data":"7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64"} Nov 03 01:31:53.178280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.177741 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85" event={"ID":"5ba736bd-c7fa-48e5-86f9-013aaea7beb7","Type":"ContainerDied","Data":"4a6428da58280e61e70919493718ac41d3603805545cf049ded74f924ad67f3b"} Nov 03 01:31:53.178280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.177766 5144 scope.go:122] "RemoveContainer" containerID="7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64" Nov 03 01:31:53.201811 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.201784 5144 scope.go:122] "RemoveContainer" containerID="7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64" Nov 03 01:31:53.202170 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:31:53.202146 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64\": container with ID starting with 7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64 not found: ID does not exist" containerID="7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64" Nov 03 01:31:53.202233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.202180 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64"} err="failed to get container status \"7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64\": rpc error: code = NotFound desc = could not find container \"7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64\": container with ID starting with 7b0e55cf52d048695f0370e5cac722ae8658494a525baee6075ef069a0c11b64 not found: ID does not exist" Nov 03 01:31:53.391488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.391457 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85"] Nov 03 01:31:53.411258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.411217 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-7b3137d3-kserve-7dd76697-8gz85"] Nov 03 01:31:53.712083 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.712050 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:31:53.803101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.803057 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca\") pod \"bc592391-fc11-4097-86b4-2b7e9290ccb7\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " Nov 03 01:31:53.803101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.803103 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca\") pod \"bc592391-fc11-4097-86b4-2b7e9290ccb7\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " Nov 03 01:31:53.803397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.803145 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7\" (UniqueName: \"kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7\") pod \"bc592391-fc11-4097-86b4-2b7e9290ccb7\" (UID: \"bc592391-fc11-4097-86b4-2b7e9290ccb7\") " Nov 03 01:31:53.803527 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.803496 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca" pod "bc592391-fc11-4097-86b4-2b7e9290ccb7" (UID: "bc592391-fc11-4097-86b4-2b7e9290ccb7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:31:53.805731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.805703 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca" pod "bc592391-fc11-4097-86b4-2b7e9290ccb7" (UID: "bc592391-fc11-4097-86b4-2b7e9290ccb7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:31:53.805731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.805723 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7" pod "bc592391-fc11-4097-86b4-2b7e9290ccb7" (UID: "bc592391-fc11-4097-86b4-2b7e9290ccb7"). InnerVolumeSpecName "kube-api-access-jsjk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:31:53.904283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.904170 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:53.904283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.904210 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsjk7\" (UniqueName: \"kubernetes.io/projected/bc592391-fc11-4097-86b4-2b7e9290ccb7-kube-api-access-jsjk7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:53.904283 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.904226 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc592391-fc11-4097-86b4-2b7e9290ccb7-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:31:53.999313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:53.999274 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ba736bd-c7fa-48e5-86f9-013aaea7beb7" path="/var/lib/kubelet/pods/5ba736bd-c7fa-48e5-86f9-013aaea7beb7/volumes" Nov 03 01:31:54.192850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:54.192327 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" event={"ID":"bc592391-fc11-4097-86b4-2b7e9290ccb7","Type":"ContainerDied","Data":"ca5c0546e0017160db65332c867555ca751eb6639333b62841a1a3812f0812cc"} Nov 03 01:31:54.192850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:54.192394 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca5c0546e0017160db65332c867555ca751eb6639333b62841a1a3812f0812cc" Nov 03 01:31:54.192850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:31:54.192473 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh" Nov 03 01:32:12.156223 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:32:12.156181 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:32:13.959668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.959606 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd"] Nov 03 01:32:13.960121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.960106 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5ba736bd-c7fa-48e5-86f9-013aaea7beb7" containerName="main" Nov 03 01:32:13.960168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.960122 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ba736bd-c7fa-48e5-86f9-013aaea7beb7" containerName="main" Nov 03 01:32:13.960168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.960154 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bc592391-fc11-4097-86b4-2b7e9290ccb7" containerName="revoke-keys" Nov 03 01:32:13.960168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.960159 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc592391-fc11-4097-86b4-2b7e9290ccb7" containerName="revoke-keys" Nov 03 01:32:13.960279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.960231 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc592391-fc11-4097-86b4-2b7e9290ccb7" containerName="revoke-keys" Nov 03 01:32:13.960279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.960243 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ba736bd-c7fa-48e5-86f9-013aaea7beb7" containerName="main" Nov 03 01:32:13.964770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:13.964748 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.018263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.018226 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd"] Nov 03 01:32:14.063933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.063893 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:32:14.063933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.063913 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:32:14.108219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.108180 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.108219 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.108217 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.108448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.108362 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdhfp\" (UniqueName: \"kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.209055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.209021 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.209055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.209061 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.209367 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.209124 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdhfp\" (UniqueName: \"kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.209912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.209847 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.211762 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.211740 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.266057 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.266022 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdhfp\" (UniqueName: \"kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp\") pod \"maas-api-revoke-sub-e15a4e09-jmfzd\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.278885 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.278854 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:14.653951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.653924 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd"] Nov 03 01:32:14.655745 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:32:14.655704 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37b5a70_18cd_4802_9f40_1f250c350006.slice/crio-32f93de1371b858c11290adf6110b851e1dfdea2f8eb7107c4b114df5f95de6e WatchSource:0}: Error finding container 32f93de1371b858c11290adf6110b851e1dfdea2f8eb7107c4b114df5f95de6e: Status 404 returned error can't find the container with id 32f93de1371b858c11290adf6110b851e1dfdea2f8eb7107c4b114df5f95de6e Nov 03 01:32:14.658096 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.658066 5144 scope.go:122] "RemoveContainer" containerID="f2f8d7556a3e7671d10af06216d0bfe20a02c91f9fe811f8cd493b398a9b9743" Nov 03 01:32:14.673148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:14.673124 5144 scope.go:122] "RemoveContainer" containerID="219ee53971c905a3007e80d7b638849e21b88618e9b9f317e943bd674ea011a5" Nov 03 01:32:15.387535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:15.387501 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" event={"ID":"c37b5a70-18cd-4802-9f40-1f250c350006","Type":"ContainerStarted","Data":"ff2452722b99a83dacf5dbdd5f869e44d7e3811fd023cb58f179ab42b39235c7"} Nov 03 01:32:15.387982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:15.387567 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" event={"ID":"c37b5a70-18cd-4802-9f40-1f250c350006","Type":"ContainerStarted","Data":"32f93de1371b858c11290adf6110b851e1dfdea2f8eb7107c4b114df5f95de6e"} Nov 03 01:32:15.500457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:15.500391 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" podStartSLOduration=2.500372024 podStartE2EDuration="2.500372024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:15.500260367 +0000 UTC m=+5225.645954827" watchObservedRunningTime="2026-11-03 01:32:15.500372024 +0000 UTC m=+5225.646066482" Nov 03 01:32:16.394766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:16.394733 5144 generic.go:356] "Generic (PLEG): container finished" podID="c37b5a70-18cd-4802-9f40-1f250c350006" containerID="ff2452722b99a83dacf5dbdd5f869e44d7e3811fd023cb58f179ab42b39235c7" exitCode=0 Nov 03 01:32:16.395200 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:16.394800 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" event={"ID":"c37b5a70-18cd-4802-9f40-1f250c350006","Type":"ContainerDied","Data":"ff2452722b99a83dacf5dbdd5f869e44d7e3811fd023cb58f179ab42b39235c7"} Nov 03 01:32:17.545145 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.545118 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:17.645574 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.645536 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca\") pod \"c37b5a70-18cd-4802-9f40-1f250c350006\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " Nov 03 01:32:17.645786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.645747 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca\") pod \"c37b5a70-18cd-4802-9f40-1f250c350006\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " Nov 03 01:32:17.645786 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.645770 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp\" (UniqueName: \"kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp\") pod \"c37b5a70-18cd-4802-9f40-1f250c350006\" (UID: \"c37b5a70-18cd-4802-9f40-1f250c350006\") " Nov 03 01:32:17.645961 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.645937 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca" pod "c37b5a70-18cd-4802-9f40-1f250c350006" (UID: "c37b5a70-18cd-4802-9f40-1f250c350006"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:32:17.648012 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.647982 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca" pod "c37b5a70-18cd-4802-9f40-1f250c350006" (UID: "c37b5a70-18cd-4802-9f40-1f250c350006"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:32:17.648104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.648006 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp" pod "c37b5a70-18cd-4802-9f40-1f250c350006" (UID: "c37b5a70-18cd-4802-9f40-1f250c350006"). InnerVolumeSpecName "kube-api-access-kdhfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:32:17.746803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.746770 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:17.746803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.746797 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdhfp\" (UniqueName: \"kubernetes.io/projected/c37b5a70-18cd-4802-9f40-1f250c350006-kube-api-access-kdhfp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:17.746803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:17.746810 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c37b5a70-18cd-4802-9f40-1f250c350006-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:18.408209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:18.408123 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" Nov 03 01:32:18.408209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:18.408153 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd" event={"ID":"c37b5a70-18cd-4802-9f40-1f250c350006","Type":"ContainerDied","Data":"32f93de1371b858c11290adf6110b851e1dfdea2f8eb7107c4b114df5f95de6e"} Nov 03 01:32:18.408209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:18.408201 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f93de1371b858c11290adf6110b851e1dfdea2f8eb7107c4b114df5f95de6e" Nov 03 01:32:30.703923 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.703885 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw"] Nov 03 01:32:30.704433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.704333 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c37b5a70-18cd-4802-9f40-1f250c350006" containerName="revoke-keys" Nov 03 01:32:30.704433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.704343 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37b5a70-18cd-4802-9f40-1f250c350006" containerName="revoke-keys" Nov 03 01:32:30.704510 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.704446 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37b5a70-18cd-4802-9f40-1f250c350006" containerName="revoke-keys" Nov 03 01:32:30.707720 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.707704 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:30.763890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.763855 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw"] Nov 03 01:32:30.778099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.778066 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume\") pod \"payload-pre-processing-6f47b87f6-b9wzw\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:30.778295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.778223 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6mst\" (UniqueName: \"kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst\") pod \"payload-pre-processing-6f47b87f6-b9wzw\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:30.879736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.879626 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6mst\" (UniqueName: \"kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst\") pod \"payload-pre-processing-6f47b87f6-b9wzw\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:30.879956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.879752 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume\") pod \"payload-pre-processing-6f47b87f6-b9wzw\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:30.880515 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.880487 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume\") pod \"payload-pre-processing-6f47b87f6-b9wzw\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:30.936519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:30.936480 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6mst\" (UniqueName: \"kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst\") pod \"payload-pre-processing-6f47b87f6-b9wzw\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:31.021294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:31.021247 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:31.396387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:31.396360 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw"] Nov 03 01:32:31.398505 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:32:31.398478 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1f82b6_344f_409c_a944_7d93c986584d.slice/crio-d9aa2e6e6d746e1e7fc656b7eb1cd83f9feacf39d898c28036e18baca0e15847 WatchSource:0}: Error finding container d9aa2e6e6d746e1e7fc656b7eb1cd83f9feacf39d898c28036e18baca0e15847: Status 404 returned error can't find the container with id d9aa2e6e6d746e1e7fc656b7eb1cd83f9feacf39d898c28036e18baca0e15847 Nov 03 01:32:31.486079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:31.486047 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" event={"ID":"bb1f82b6-344f-409c-a944-7d93c986584d","Type":"ContainerStarted","Data":"d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428"} Nov 03 01:32:31.486296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:31.486105 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" event={"ID":"bb1f82b6-344f-409c-a944-7d93c986584d","Type":"ContainerStarted","Data":"d9aa2e6e6d746e1e7fc656b7eb1cd83f9feacf39d898c28036e18baca0e15847"} Nov 03 01:32:31.486296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:31.486149 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:31.600651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:31.600569 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" podStartSLOduration=1.600552962 podStartE2EDuration="1.600552962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:31.598677877 +0000 UTC m=+5241.744372358" watchObservedRunningTime="2026-11-03 01:32:31.600552962 +0000 UTC m=+5241.746247419" Nov 03 01:32:42.156289 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:32:42.156252 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:32:42.495420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:42.495338 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:32:42.660979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:42.660939 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn"] Nov 03 01:32:44.655331 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.655297 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b99c9ccc8-7hwwn"] Nov 03 01:32:44.659274 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.659258 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.714448 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.714404 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b99c9ccc8-7hwwn"] Nov 03 01:32:44.720206 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.720175 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgn7b\" (UniqueName: \"kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b\") pod \"payload-processing-5b99c9ccc8-7hwwn\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.720383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.720244 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume\") pod \"payload-processing-5b99c9ccc8-7hwwn\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.821702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.821664 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgn7b\" (UniqueName: \"kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b\") pod \"payload-processing-5b99c9ccc8-7hwwn\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.821921 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.821729 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume\") pod \"payload-processing-5b99c9ccc8-7hwwn\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.822412 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.822383 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume\") pod \"payload-processing-5b99c9ccc8-7hwwn\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.878168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.878136 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgn7b\" (UniqueName: \"kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b\") pod \"payload-processing-5b99c9ccc8-7hwwn\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:44.972399 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:44.972314 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:45.346380 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:45.346351 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b99c9ccc8-7hwwn"] Nov 03 01:32:45.348295 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:32:45.348266 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32f0c1b_c02b_43df_8c3b_31f8bc9d37e4.slice/crio-7745c4e6f28a70e512d20c31e26ee88f32e521f718b8aef7fbdd61d37407ab3a WatchSource:0}: Error finding container 7745c4e6f28a70e512d20c31e26ee88f32e521f718b8aef7fbdd61d37407ab3a: Status 404 returned error can't find the container with id 7745c4e6f28a70e512d20c31e26ee88f32e521f718b8aef7fbdd61d37407ab3a Nov 03 01:32:45.582126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:45.582088 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" event={"ID":"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4","Type":"ContainerStarted","Data":"1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28"} Nov 03 01:32:45.582317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:45.582147 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" event={"ID":"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4","Type":"ContainerStarted","Data":"7745c4e6f28a70e512d20c31e26ee88f32e521f718b8aef7fbdd61d37407ab3a"} Nov 03 01:32:45.582317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:45.582169 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:45.696378 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:45.696268 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" podStartSLOduration=1.6962535330000001 podStartE2EDuration="1.696253533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:45.694939387 +0000 UTC m=+5255.840633869" watchObservedRunningTime="2026-11-03 01:32:45.696253533 +0000 UTC m=+5255.841947990" Nov 03 01:32:47.661749 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:47.661706 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" podUID="83ca4574-d201-4912-9a0a-ddaadb5d598e" containerName="payload-pre-processing" containerID="cri-o://65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e" gracePeriod=25 Nov 03 01:32:47.927661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:47.927575 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 01:32:48.064118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.056102 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5\" (UniqueName: \"kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5\") pod \"83ca4574-d201-4912-9a0a-ddaadb5d598e\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " Nov 03 01:32:48.064118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.056177 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume\") pod \"83ca4574-d201-4912-9a0a-ddaadb5d598e\" (UID: \"83ca4574-d201-4912-9a0a-ddaadb5d598e\") " Nov 03 01:32:48.064118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.063845 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume" pod "83ca4574-d201-4912-9a0a-ddaadb5d598e" (UID: "83ca4574-d201-4912-9a0a-ddaadb5d598e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:32:48.066382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.066334 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5" pod "83ca4574-d201-4912-9a0a-ddaadb5d598e" (UID: "83ca4574-d201-4912-9a0a-ddaadb5d598e"). InnerVolumeSpecName "kube-api-access-6vpp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:32:48.157184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.157151 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vpp5\" (UniqueName: \"kubernetes.io/projected/83ca4574-d201-4912-9a0a-ddaadb5d598e-kube-api-access-6vpp5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:48.157184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.157177 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83ca4574-d201-4912-9a0a-ddaadb5d598e-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:32:48.601827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.601785 5144 generic.go:356] "Generic (PLEG): container finished" podID="83ca4574-d201-4912-9a0a-ddaadb5d598e" containerID="65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e" exitCode=0 Nov 03 01:32:48.602058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.601838 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" Nov 03 01:32:48.602058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.601871 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" event={"ID":"83ca4574-d201-4912-9a0a-ddaadb5d598e","Type":"ContainerDied","Data":"65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e"} Nov 03 01:32:48.602058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.601938 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn" event={"ID":"83ca4574-d201-4912-9a0a-ddaadb5d598e","Type":"ContainerDied","Data":"ae2b637319d2df00208428c88206cbd4f4d9beef1d9bfc35199269cc50ff81a3"} Nov 03 01:32:48.602058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.601970 5144 scope.go:122] "RemoveContainer" containerID="65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e" Nov 03 01:32:48.611963 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.611940 5144 scope.go:122] "RemoveContainer" containerID="65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e" Nov 03 01:32:48.612239 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:32:48.612217 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e\": container with ID starting with 65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e not found: ID does not exist" containerID="65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e" Nov 03 01:32:48.612307 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.612248 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e"} err="failed to get container status \"65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e\": rpc error: code = NotFound desc = could not find container \"65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e\": container with ID starting with 65bfdfdf74aac4d61a31e8d3e26344500ffcf6d1e87b0f79c2a8d266c2a4172e not found: ID does not exist" Nov 03 01:32:48.765348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.765318 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn"] Nov 03 01:32:48.814426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:48.814392 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-w6pcn"] Nov 03 01:32:49.994823 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:49.994789 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83ca4574-d201-4912-9a0a-ddaadb5d598e" path="/var/lib/kubelet/pods/83ca4574-d201-4912-9a0a-ddaadb5d598e/volumes" Nov 03 01:32:56.592552 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:56.592521 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:32:56.748140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:56.748101 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vh6mt"] Nov 03 01:32:58.876013 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:58.875979 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-ccd6f9449-lp87k"] Nov 03 01:32:58.876525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:58.876501 5144 cpu_manager.go:414] "containerMap: removing container" podUID="83ca4574-d201-4912-9a0a-ddaadb5d598e" containerName="payload-pre-processing" Nov 03 01:32:58.876586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:58.876528 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83ca4574-d201-4912-9a0a-ddaadb5d598e" containerName="payload-pre-processing" Nov 03 01:32:58.876621 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:58.876615 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="83ca4574-d201-4912-9a0a-ddaadb5d598e" containerName="payload-pre-processing" Nov 03 01:32:58.879838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:58.879819 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:58.946161 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:58.946124 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-ccd6f9449-lp87k"] Nov 03 01:32:59.074043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.074001 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2b6a92-c4bb-40a2-a630-ae33f705e854-plugins-config-volume\") pod \"payload-processing-ccd6f9449-lp87k\" (UID: \"5e2b6a92-c4bb-40a2-a630-ae33f705e854\") " pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.074247 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.074059 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkbx7\" (UniqueName: \"kubernetes.io/projected/5e2b6a92-c4bb-40a2-a630-ae33f705e854-kube-api-access-lkbx7\") pod \"payload-processing-ccd6f9449-lp87k\" (UID: \"5e2b6a92-c4bb-40a2-a630-ae33f705e854\") " pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.174897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.174805 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2b6a92-c4bb-40a2-a630-ae33f705e854-plugins-config-volume\") pod \"payload-processing-ccd6f9449-lp87k\" (UID: \"5e2b6a92-c4bb-40a2-a630-ae33f705e854\") " pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.174897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.174860 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkbx7\" (UniqueName: \"kubernetes.io/projected/5e2b6a92-c4bb-40a2-a630-ae33f705e854-kube-api-access-lkbx7\") pod \"payload-processing-ccd6f9449-lp87k\" (UID: \"5e2b6a92-c4bb-40a2-a630-ae33f705e854\") " pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.175496 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.175473 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e2b6a92-c4bb-40a2-a630-ae33f705e854-plugins-config-volume\") pod \"payload-processing-ccd6f9449-lp87k\" (UID: \"5e2b6a92-c4bb-40a2-a630-ae33f705e854\") " pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.231864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.231830 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkbx7\" (UniqueName: \"kubernetes.io/projected/5e2b6a92-c4bb-40a2-a630-ae33f705e854-kube-api-access-lkbx7\") pod \"payload-processing-ccd6f9449-lp87k\" (UID: \"5e2b6a92-c4bb-40a2-a630-ae33f705e854\") " pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.493826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.493733 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:32:59.865881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:32:59.865852 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-ccd6f9449-lp87k"] Nov 03 01:32:59.867309 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:32:59.867272 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e2b6a92_c4bb_40a2_a630_ae33f705e854.slice/crio-b68ad4946c46528c3848a6ba5f5a0467fe3668efb4ee6cabaf492dc604831f92 WatchSource:0}: Error finding container b68ad4946c46528c3848a6ba5f5a0467fe3668efb4ee6cabaf492dc604831f92: Status 404 returned error can't find the container with id b68ad4946c46528c3848a6ba5f5a0467fe3668efb4ee6cabaf492dc604831f92 Nov 03 01:33:00.687972 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:00.687923 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" event={"ID":"5e2b6a92-c4bb-40a2-a630-ae33f705e854","Type":"ContainerStarted","Data":"5d3dac34b14459e0e2e5268f151c7fdd05a62c05dc2edd6694cf87f1959029bc"} Nov 03 01:33:00.688386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:00.687996 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:33:00.688386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:00.688019 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" event={"ID":"5e2b6a92-c4bb-40a2-a630-ae33f705e854","Type":"ContainerStarted","Data":"b68ad4946c46528c3848a6ba5f5a0467fe3668efb4ee6cabaf492dc604831f92"} Nov 03 01:33:00.804768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:00.804712 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" podStartSLOduration=2.804695476 podStartE2EDuration="2.804695476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:00.804421943 +0000 UTC m=+5270.950116400" watchObservedRunningTime="2026-11-03 01:33:00.804695476 +0000 UTC m=+5270.950389934" Nov 03 01:33:01.748828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:01.748757 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" podUID="308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" containerName="payload-processing" containerID="cri-o://82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3" gracePeriod=25 Nov 03 01:33:01.997012 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:01.996985 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 01:33:02.000350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.000302 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume\") pod \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " Nov 03 01:33:02.000425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.000347 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m\" (UniqueName: \"kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m\") pod \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\" (UID: \"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1\") " Nov 03 01:33:02.000744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.000722 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume" pod "308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" (UID: "308a18b8-bcee-45dc-b1e5-4b10fcabc3c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:02.002700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.002676 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m" pod "308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" (UID: "308a18b8-bcee-45dc-b1e5-4b10fcabc3c1"). InnerVolumeSpecName "kube-api-access-m7c7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:02.101922 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.101884 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:33:02.101922 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.101915 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7c7m\" (UniqueName: \"kubernetes.io/projected/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1-kube-api-access-m7c7m\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:33:02.701260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.701222 5144 generic.go:356] "Generic (PLEG): container finished" podID="308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" containerID="82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3" exitCode=0 Nov 03 01:33:02.701468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.701275 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" Nov 03 01:33:02.701468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.701282 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" event={"ID":"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1","Type":"ContainerDied","Data":"82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3"} Nov 03 01:33:02.701468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.701323 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-vh6mt" event={"ID":"308a18b8-bcee-45dc-b1e5-4b10fcabc3c1","Type":"ContainerDied","Data":"cef42142fef73755da52c6f1eaade348f3ab767001993fc702e7e59f1c1ac861"} Nov 03 01:33:02.701468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.701344 5144 scope.go:122] "RemoveContainer" containerID="82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3" Nov 03 01:33:02.712811 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.712792 5144 scope.go:122] "RemoveContainer" containerID="82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3" Nov 03 01:33:02.713096 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:33:02.713075 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3\": container with ID starting with 82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3 not found: ID does not exist" containerID="82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3" Nov 03 01:33:02.713180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.713107 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3"} err="failed to get container status \"82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3\": rpc error: code = NotFound desc = could not find container \"82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3\": container with ID starting with 82f93dd0303c48ddbbec1b6d4a6b038a745c09d932a1604a48388e843628e5b3 not found: ID does not exist" Nov 03 01:33:02.864176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.864135 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vh6mt"] Nov 03 01:33:02.913508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:02.913473 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-vh6mt"] Nov 03 01:33:03.994908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:03.994876 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" path="/var/lib/kubelet/pods/308a18b8-bcee-45dc-b1e5-4b10fcabc3c1/volumes" Nov 03 01:33:10.975132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:10.975100 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw"] Nov 03 01:33:11.697686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:11.697659 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-ccd6f9449-lp87k" Nov 03 01:33:11.833036 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:11.833002 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b99c9ccc8-7hwwn"] Nov 03 01:33:12.154909 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:33:12.154868 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:33:14.708921 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:14.708887 5144 scope.go:122] "RemoveContainer" containerID="075b3e8d8f9f42a90e4125a035d4f85ee158c85230cacf031ef42bc426aa4c96" Nov 03 01:33:15.976322 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:15.976268 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" podUID="bb1f82b6-344f-409c-a944-7d93c986584d" containerName="payload-pre-processing" containerID="cri-o://d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428" gracePeriod=25 Nov 03 01:33:16.246881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.246846 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:33:16.352704 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.352660 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst\" (UniqueName: \"kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst\") pod \"bb1f82b6-344f-409c-a944-7d93c986584d\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " Nov 03 01:33:16.352888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.352825 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume\") pod \"bb1f82b6-344f-409c-a944-7d93c986584d\" (UID: \"bb1f82b6-344f-409c-a944-7d93c986584d\") " Nov 03 01:33:16.353220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.353192 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume" pod "bb1f82b6-344f-409c-a944-7d93c986584d" (UID: "bb1f82b6-344f-409c-a944-7d93c986584d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:16.354941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.354920 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst" pod "bb1f82b6-344f-409c-a944-7d93c986584d" (UID: "bb1f82b6-344f-409c-a944-7d93c986584d"). InnerVolumeSpecName "kube-api-access-t6mst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:16.453711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.453678 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb1f82b6-344f-409c-a944-7d93c986584d-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:33:16.453711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.453707 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6mst\" (UniqueName: \"kubernetes.io/projected/bb1f82b6-344f-409c-a944-7d93c986584d-kube-api-access-t6mst\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:33:16.791618 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.791581 5144 generic.go:356] "Generic (PLEG): container finished" podID="bb1f82b6-344f-409c-a944-7d93c986584d" containerID="d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428" exitCode=0 Nov 03 01:33:16.791850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.791642 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" Nov 03 01:33:16.791850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.791651 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" event={"ID":"bb1f82b6-344f-409c-a944-7d93c986584d","Type":"ContainerDied","Data":"d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428"} Nov 03 01:33:16.791850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.791735 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw" event={"ID":"bb1f82b6-344f-409c-a944-7d93c986584d","Type":"ContainerDied","Data":"d9aa2e6e6d746e1e7fc656b7eb1cd83f9feacf39d898c28036e18baca0e15847"} Nov 03 01:33:16.791850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.791765 5144 scope.go:122] "RemoveContainer" containerID="d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428" Nov 03 01:33:16.801913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.801893 5144 scope.go:122] "RemoveContainer" containerID="d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428" Nov 03 01:33:16.802188 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:33:16.802166 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428\": container with ID starting with d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428 not found: ID does not exist" containerID="d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428" Nov 03 01:33:16.802248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.802195 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428"} err="failed to get container status \"d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428\": rpc error: code = NotFound desc = could not find container \"d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428\": container with ID starting with d9b1b836ec53122a65b33bae6994aa4f776fadd88c9b9903ebf04249c0b39428 not found: ID does not exist" Nov 03 01:33:16.833662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.833580 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" podUID="e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" containerName="payload-processing" containerID="cri-o://1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28" gracePeriod=25 Nov 03 01:33:16.954779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.954729 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw"] Nov 03 01:33:16.957699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:16.957670 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-6f47b87f6-b9wzw"] Nov 03 01:33:17.100323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.100288 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:33:17.160348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.160307 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume\") pod \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " Nov 03 01:33:17.160573 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.160372 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b\" (UniqueName: \"kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b\") pod \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\" (UID: \"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4\") " Nov 03 01:33:17.160837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.160813 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume" pod "e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" (UID: "e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:33:17.162814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.162785 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b" pod "e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" (UID: "e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4"). InnerVolumeSpecName "kube-api-access-mgn7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:33:17.261288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.261246 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:33:17.261288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.261285 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgn7b\" (UniqueName: \"kubernetes.io/projected/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4-kube-api-access-mgn7b\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:33:17.804868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.804822 5144 generic.go:356] "Generic (PLEG): container finished" podID="e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" containerID="1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28" exitCode=0 Nov 03 01:33:17.804868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.804877 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" Nov 03 01:33:17.805255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.804883 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" event={"ID":"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4","Type":"ContainerDied","Data":"1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28"} Nov 03 01:33:17.805255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.804957 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b99c9ccc8-7hwwn" event={"ID":"e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4","Type":"ContainerDied","Data":"7745c4e6f28a70e512d20c31e26ee88f32e521f718b8aef7fbdd61d37407ab3a"} Nov 03 01:33:17.805255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.804991 5144 scope.go:122] "RemoveContainer" containerID="1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28" Nov 03 01:33:17.823730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.823703 5144 scope.go:122] "RemoveContainer" containerID="1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28" Nov 03 01:33:17.824118 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:33:17.824097 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28\": container with ID starting with 1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28 not found: ID does not exist" containerID="1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28" Nov 03 01:33:17.824211 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.824127 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28"} err="failed to get container status \"1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28\": rpc error: code = NotFound desc = could not find container \"1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28\": container with ID starting with 1ab13a864cb21f31548b5b363f58dc6706d4f06199f68fe9a1a062ec9e444a28 not found: ID does not exist" Nov 03 01:33:17.967141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.967105 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-5b99c9ccc8-7hwwn"] Nov 03 01:33:17.996151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:17.996111 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb1f82b6-344f-409c-a944-7d93c986584d" path="/var/lib/kubelet/pods/bb1f82b6-344f-409c-a944-7d93c986584d/volumes" Nov 03 01:33:18.016533 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.016492 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-5b99c9ccc8-7hwwn"] Nov 03 01:33:18.257522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.257483 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c"] Nov 03 01:33:18.258376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258336 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bb1f82b6-344f-409c-a944-7d93c986584d" containerName="payload-pre-processing" Nov 03 01:33:18.258512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258376 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb1f82b6-344f-409c-a944-7d93c986584d" containerName="payload-pre-processing" Nov 03 01:33:18.258512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258442 5144 cpu_manager.go:414] "containerMap: removing container" podUID="308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" containerName="payload-processing" Nov 03 01:33:18.258512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258451 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" containerName="payload-processing" Nov 03 01:33:18.258512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258469 5144 cpu_manager.go:414] "containerMap: removing container" podUID="e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" containerName="payload-processing" Nov 03 01:33:18.258512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258476 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" containerName="payload-processing" Nov 03 01:33:18.258817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258594 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="308a18b8-bcee-45dc-b1e5-4b10fcabc3c1" containerName="payload-processing" Nov 03 01:33:18.258817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258609 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" containerName="payload-processing" Nov 03 01:33:18.258817 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.258621 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb1f82b6-344f-409c-a944-7d93c986584d" containerName="payload-pre-processing" Nov 03 01:33:18.263221 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.263193 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.316039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.315828 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\"" Nov 03 01:33:18.316039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.315874 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\"" Nov 03 01:33:18.316039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.315924 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\"" Nov 03 01:33:18.316039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.315943 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\"" Nov 03 01:33:18.316039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.316000 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\"" Nov 03 01:33:18.318012 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.317988 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c"] Nov 03 01:33:18.373008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.372967 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.373207 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.373081 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vdvl\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.373207 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.373143 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.373285 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.373211 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.474135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.474092 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.474135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.474138 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.474410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.474217 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.474410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.474247 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vdvl\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.475055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.475002 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.475207 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.475187 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.477026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.477003 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.531265 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.531191 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vdvl\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl\") pod \"payload-processing-external-model-5bcd7997-8rx8c\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.578229 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.578182 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:18.962937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:18.962899 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c"] Nov 03 01:33:18.964046 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:33:18.964005 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19135338_6f2b_487b_a962_f1130e456eca.slice/crio-f764d09c0a99e42599c3577494419b529cea325e2015c7476466782045b848ce WatchSource:0}: Error finding container f764d09c0a99e42599c3577494419b529cea325e2015c7476466782045b848ce: Status 404 returned error can't find the container with id f764d09c0a99e42599c3577494419b529cea325e2015c7476466782045b848ce Nov 03 01:33:19.823169 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:19.823132 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" event={"ID":"19135338-6f2b-487b-a962-f1130e456eca","Type":"ContainerStarted","Data":"1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee"} Nov 03 01:33:19.823689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:19.823190 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" event={"ID":"19135338-6f2b-487b-a962-f1130e456eca","Type":"ContainerStarted","Data":"f764d09c0a99e42599c3577494419b529cea325e2015c7476466782045b848ce"} Nov 03 01:33:19.823689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:19.823253 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:19.936925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:19.936857 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" podStartSLOduration=1.936841839 podStartE2EDuration="1.936841839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:19.935240285 +0000 UTC m=+5290.080934743" watchObservedRunningTime="2026-11-03 01:33:19.936841839 +0000 UTC m=+5290.082536292" Nov 03 01:33:19.996532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:19.996495 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4" path="/var/lib/kubelet/pods/e32f0c1b-c02b-43df-8c3b-31f8bc9d37e4/volumes" Nov 03 01:33:21.998846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:21.998756 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:33:21.999373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:21.998888 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:33:30.837195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:33:30.837158 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:33:42.155400 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:33:42.155355 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:34:12.156929 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:34:12.156889 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:34:19.044567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:19.044469 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx"] Nov 03 01:34:19.093935 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:19.093895 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b1ce5a3-r5tpx"] Nov 03 01:34:19.994921 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:19.994880 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="110558df-044b-435c-b088-ce718e0cf8be" path="/var/lib/kubelet/pods/110558df-044b-435c-b088-ce718e0cf8be/volumes" Nov 03 01:34:42.155798 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:34:42.155761 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:34:56.374704 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.374661 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5"] Nov 03 01:34:56.378410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.378389 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.433934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.433895 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5"] Nov 03 01:34:56.479779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.479740 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:34:56.479983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.479886 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:34:56.498459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.498421 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.498619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.498488 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzf78\" (UniqueName: \"kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.498744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.498605 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.599417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.599378 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzf78\" (UniqueName: \"kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.599417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.599429 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.599733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.599565 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.600316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.600251 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.602053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.602030 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.656199 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.656117 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzf78\" (UniqueName: \"kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78\") pod \"maas-api-revoke-sub-e2547622-5nlz5\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:56.692962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:56.692924 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:57.067854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:57.067828 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5"] Nov 03 01:34:57.069887 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:34:57.069855 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c146869_7021_4b25_b319_10fc2f968b82.slice/crio-ea75d26c5535306fb02972b5e4bb75a1d480d7356ffb2d79174c9abdf65150ea WatchSource:0}: Error finding container ea75d26c5535306fb02972b5e4bb75a1d480d7356ffb2d79174c9abdf65150ea: Status 404 returned error can't find the container with id ea75d26c5535306fb02972b5e4bb75a1d480d7356ffb2d79174c9abdf65150ea Nov 03 01:34:57.419345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:57.419290 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" event={"ID":"7c146869-7021-4b25-b319-10fc2f968b82","Type":"ContainerStarted","Data":"38e1ec298023838647eeeb1387e083bc87b58a9a90168e6757a11fdf4f4dd64d"} Nov 03 01:34:57.419906 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:57.419390 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" event={"ID":"7c146869-7021-4b25-b319-10fc2f968b82","Type":"ContainerStarted","Data":"ea75d26c5535306fb02972b5e4bb75a1d480d7356ffb2d79174c9abdf65150ea"} Nov 03 01:34:57.532370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:57.532309 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" podStartSLOduration=1.532295973 podStartE2EDuration="1.532295973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:34:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:57.53081961 +0000 UTC m=+5387.676514081" watchObservedRunningTime="2026-11-03 01:34:57.532295973 +0000 UTC m=+5387.677990425" Nov 03 01:34:58.427356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:58.427260 5144 generic.go:356] "Generic (PLEG): container finished" podID="7c146869-7021-4b25-b319-10fc2f968b82" containerID="38e1ec298023838647eeeb1387e083bc87b58a9a90168e6757a11fdf4f4dd64d" exitCode=0 Nov 03 01:34:58.427356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:58.427344 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" event={"ID":"7c146869-7021-4b25-b319-10fc2f968b82","Type":"ContainerDied","Data":"38e1ec298023838647eeeb1387e083bc87b58a9a90168e6757a11fdf4f4dd64d"} Nov 03 01:34:59.572880 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.572852 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:34:59.629734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.629701 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca\") pod \"7c146869-7021-4b25-b319-10fc2f968b82\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " Nov 03 01:34:59.629942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.629838 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca\") pod \"7c146869-7021-4b25-b319-10fc2f968b82\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " Nov 03 01:34:59.629942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.629884 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78\" (UniqueName: \"kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78\") pod \"7c146869-7021-4b25-b319-10fc2f968b82\" (UID: \"7c146869-7021-4b25-b319-10fc2f968b82\") " Nov 03 01:34:59.630216 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.630193 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca" pod "7c146869-7021-4b25-b319-10fc2f968b82" (UID: "7c146869-7021-4b25-b319-10fc2f968b82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:34:59.632255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.632226 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca" pod "7c146869-7021-4b25-b319-10fc2f968b82" (UID: "7c146869-7021-4b25-b319-10fc2f968b82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:34:59.632380 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.632313 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78" pod "7c146869-7021-4b25-b319-10fc2f968b82" (UID: "7c146869-7021-4b25-b319-10fc2f968b82"). InnerVolumeSpecName "kube-api-access-bzf78". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:34:59.731389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.731289 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:34:59.731389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.731325 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzf78\" (UniqueName: \"kubernetes.io/projected/7c146869-7021-4b25-b319-10fc2f968b82-kube-api-access-bzf78\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:34:59.731389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:34:59.731340 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c146869-7021-4b25-b319-10fc2f968b82-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:00.439543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:00.439440 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" Nov 03 01:35:00.439543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:00.439469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5" event={"ID":"7c146869-7021-4b25-b319-10fc2f968b82","Type":"ContainerDied","Data":"ea75d26c5535306fb02972b5e4bb75a1d480d7356ffb2d79174c9abdf65150ea"} Nov 03 01:35:00.439543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:00.439514 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea75d26c5535306fb02972b5e4bb75a1d480d7356ffb2d79174c9abdf65150ea" Nov 03 01:35:07.897343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:07.897305 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c"] Nov 03 01:35:11.249778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:11.249742 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:35:11.261602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:11.261575 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:35:12.307056 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:35:12.307008 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:35:12.897751 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:12.897592 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" podUID="19135338-6f2b-487b-a962-f1130e456eca" containerName="payload-processing" containerID="cri-o://1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee" gracePeriod=25 Nov 03 01:35:13.289826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.289799 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:35:13.309042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.309004 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume\") pod \"19135338-6f2b-487b-a962-f1130e456eca\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " Nov 03 01:35:13.309540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.309050 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl\") pod \"19135338-6f2b-487b-a962-f1130e456eca\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " Nov 03 01:35:13.309540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.309090 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials\") pod \"19135338-6f2b-487b-a962-f1130e456eca\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " Nov 03 01:35:13.309540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.309127 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume\") pod \"19135338-6f2b-487b-a962-f1130e456eca\" (UID: \"19135338-6f2b-487b-a962-f1130e456eca\") " Nov 03 01:35:13.309540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.309465 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume" pod "19135338-6f2b-487b-a962-f1130e456eca" (UID: "19135338-6f2b-487b-a962-f1130e456eca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:13.309540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.309528 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume" pod "19135338-6f2b-487b-a962-f1130e456eca" (UID: "19135338-6f2b-487b-a962-f1130e456eca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:13.311525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.311487 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl" pod "19135338-6f2b-487b-a962-f1130e456eca" (UID: "19135338-6f2b-487b-a962-f1130e456eca"). InnerVolumeSpecName "kube-api-access-2vdvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:13.311611 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.311536 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials" pod "19135338-6f2b-487b-a962-f1130e456eca" (UID: "19135338-6f2b-487b-a962-f1130e456eca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:13.410095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.410052 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.410095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.410097 5144 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/19135338-6f2b-487b-a962-f1130e456eca-routing-overlay-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.410422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.410113 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vdvl\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-kube-api-access-2vdvl\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.410422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.410128 5144 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/19135338-6f2b-487b-a962-f1130e456eca-provider-credentials\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:13.578382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.578324 5144 generic.go:356] "Generic (PLEG): container finished" podID="19135338-6f2b-487b-a962-f1130e456eca" containerID="1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee" exitCode=0 Nov 03 01:35:13.578623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.578419 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" Nov 03 01:35:13.578623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.578419 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" event={"ID":"19135338-6f2b-487b-a962-f1130e456eca","Type":"ContainerDied","Data":"1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee"} Nov 03 01:35:13.578623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.578490 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c" event={"ID":"19135338-6f2b-487b-a962-f1130e456eca","Type":"ContainerDied","Data":"f764d09c0a99e42599c3577494419b529cea325e2015c7476466782045b848ce"} Nov 03 01:35:13.578623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.578524 5144 scope.go:122] "RemoveContainer" containerID="1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee" Nov 03 01:35:13.596433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.596407 5144 scope.go:122] "RemoveContainer" containerID="1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee" Nov 03 01:35:13.596843 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:35:13.596820 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee\": container with ID starting with 1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee not found: ID does not exist" containerID="1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee" Nov 03 01:35:13.596925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.596853 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee"} err="failed to get container status \"1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee\": rpc error: code = NotFound desc = could not find container \"1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee\": container with ID starting with 1979c445adc2679cac2cc766beab1effe42482ba8a4edd17775188460e96d0ee not found: ID does not exist" Nov 03 01:35:13.741423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.741389 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c"] Nov 03 01:35:13.743819 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.743798 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-8rx8c"] Nov 03 01:35:13.995287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:13.995206 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19135338-6f2b-487b-a962-f1130e456eca" path="/var/lib/kubelet/pods/19135338-6f2b-487b-a962-f1130e456eca/volumes" Nov 03 01:35:27.220833 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.220792 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk"] Nov 03 01:35:27.221369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.221351 5144 cpu_manager.go:414] "containerMap: removing container" podUID="19135338-6f2b-487b-a962-f1130e456eca" containerName="payload-processing" Nov 03 01:35:27.221423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.221371 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19135338-6f2b-487b-a962-f1130e456eca" containerName="payload-processing" Nov 03 01:35:27.221423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.221400 5144 cpu_manager.go:414] "containerMap: removing container" podUID="7c146869-7021-4b25-b319-10fc2f968b82" containerName="revoke-keys" Nov 03 01:35:27.221423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.221405 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c146869-7021-4b25-b319-10fc2f968b82" containerName="revoke-keys" Nov 03 01:35:27.221537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.221471 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c146869-7021-4b25-b319-10fc2f968b82" containerName="revoke-keys" Nov 03 01:35:27.221537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.221482 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="19135338-6f2b-487b-a962-f1130e456eca" containerName="payload-processing" Nov 03 01:35:27.224504 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.224488 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.276607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.276571 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:35:27.280554 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.280514 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk"] Nov 03 01:35:27.316131 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.316086 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46d94\" (UniqueName: \"kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.316351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.316235 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.316351 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.316298 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.325982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.325941 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:35:27.417662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.417609 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.417936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.417675 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.417936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.417740 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46d94\" (UniqueName: \"kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.418402 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.418378 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.420285 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.420260 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.474563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.474483 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46d94\" (UniqueName: \"kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94\") pod \"maas-api-revoke-sub-c1e271a9-qm8mk\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.539485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.539447 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:27.915746 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:27.915715 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk"] Nov 03 01:35:27.917590 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:35:27.917559 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc247be5d_763f_4594_8b64_f5ecf51a3049.slice/crio-de4f84427b1f9d7742d36c4a695f88a6457e72f0eae40973346cb8278c1c5b4b WatchSource:0}: Error finding container de4f84427b1f9d7742d36c4a695f88a6457e72f0eae40973346cb8278c1c5b4b: Status 404 returned error can't find the container with id de4f84427b1f9d7742d36c4a695f88a6457e72f0eae40973346cb8278c1c5b4b Nov 03 01:35:28.695768 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:28.695725 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" event={"ID":"c247be5d-763f-4594-8b64-f5ecf51a3049","Type":"ContainerStarted","Data":"c4923f2d79750f1716c30b0ecf1e69d63d2b6042d7f3279497aaf6677806dcaf"} Nov 03 01:35:28.696188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:28.695782 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" event={"ID":"c247be5d-763f-4594-8b64-f5ecf51a3049","Type":"ContainerStarted","Data":"de4f84427b1f9d7742d36c4a695f88a6457e72f0eae40973346cb8278c1c5b4b"} Nov 03 01:35:28.809614 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:28.809545 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" podStartSLOduration=1.809528528 podStartE2EDuration="1.809528528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:35:28.806792609 +0000 UTC m=+5418.952487066" watchObservedRunningTime="2026-11-03 01:35:28.809528528 +0000 UTC m=+5418.955222986" Nov 03 01:35:29.702873 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:29.702836 5144 generic.go:356] "Generic (PLEG): container finished" podID="c247be5d-763f-4594-8b64-f5ecf51a3049" containerID="c4923f2d79750f1716c30b0ecf1e69d63d2b6042d7f3279497aaf6677806dcaf" exitCode=0 Nov 03 01:35:29.703331 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:29.702926 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" event={"ID":"c247be5d-763f-4594-8b64-f5ecf51a3049","Type":"ContainerDied","Data":"c4923f2d79750f1716c30b0ecf1e69d63d2b6042d7f3279497aaf6677806dcaf"} Nov 03 01:35:30.856031 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.856006 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:30.957050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.957015 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca\") pod \"c247be5d-763f-4594-8b64-f5ecf51a3049\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " Nov 03 01:35:30.957235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.957118 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca\") pod \"c247be5d-763f-4594-8b64-f5ecf51a3049\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " Nov 03 01:35:30.957235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.957147 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94\" (UniqueName: \"kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94\") pod \"c247be5d-763f-4594-8b64-f5ecf51a3049\" (UID: \"c247be5d-763f-4594-8b64-f5ecf51a3049\") " Nov 03 01:35:30.957492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.957462 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca" pod "c247be5d-763f-4594-8b64-f5ecf51a3049" (UID: "c247be5d-763f-4594-8b64-f5ecf51a3049"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:30.959461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.959394 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca" pod "c247be5d-763f-4594-8b64-f5ecf51a3049" (UID: "c247be5d-763f-4594-8b64-f5ecf51a3049"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:30.959572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:30.959509 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94" pod "c247be5d-763f-4594-8b64-f5ecf51a3049" (UID: "c247be5d-763f-4594-8b64-f5ecf51a3049"). InnerVolumeSpecName "kube-api-access-46d94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:31.058778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:31.058738 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:31.058778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:31.058772 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46d94\" (UniqueName: \"kubernetes.io/projected/c247be5d-763f-4594-8b64-f5ecf51a3049-kube-api-access-46d94\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:31.058778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:31.058782 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c247be5d-763f-4594-8b64-f5ecf51a3049-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:31.716700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:31.716657 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" event={"ID":"c247be5d-763f-4594-8b64-f5ecf51a3049","Type":"ContainerDied","Data":"de4f84427b1f9d7742d36c4a695f88a6457e72f0eae40973346cb8278c1c5b4b"} Nov 03 01:35:31.716700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:31.716690 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk" Nov 03 01:35:31.716988 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:31.716746 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de4f84427b1f9d7742d36c4a695f88a6457e72f0eae40973346cb8278c1c5b4b" Nov 03 01:35:42.157664 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:35:42.157597 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:35:51.971741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:51.971704 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq"] Nov 03 01:35:51.972202 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:51.972185 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c247be5d-763f-4594-8b64-f5ecf51a3049" containerName="revoke-keys" Nov 03 01:35:51.972250 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:51.972206 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c247be5d-763f-4594-8b64-f5ecf51a3049" containerName="revoke-keys" Nov 03 01:35:51.972311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:51.972301 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c247be5d-763f-4594-8b64-f5ecf51a3049" containerName="revoke-keys" Nov 03 01:35:51.975651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:51.975608 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.027896 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.027864 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:35:52.031589 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.031558 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq"] Nov 03 01:35:52.077299 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.077270 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:35:52.098083 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.098051 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.098300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.098273 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr2km\" (UniqueName: \"kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.098513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.098488 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.199766 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.199716 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr2km\" (UniqueName: \"kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.199969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.199852 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.199969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.199932 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.200488 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.200467 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.202541 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.202514 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.256948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.256914 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr2km\" (UniqueName: \"kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km\") pod \"maas-api-revoke-sub-2e1f5b5f-n96rq\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.293031 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.292999 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:52.668388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.668362 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq"] Nov 03 01:35:52.670545 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:35:52.670517 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70ac437_31a0_41ac_be02_af0aa2659d62.slice/crio-3b02cf028c881fe72942d0933b3a66de4815b8730343e162ac929b30968f51b2 WatchSource:0}: Error finding container 3b02cf028c881fe72942d0933b3a66de4815b8730343e162ac929b30968f51b2: Status 404 returned error can't find the container with id 3b02cf028c881fe72942d0933b3a66de4815b8730343e162ac929b30968f51b2 Nov 03 01:35:52.866995 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:52.866963 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" event={"ID":"c70ac437-31a0-41ac-be02-af0aa2659d62","Type":"ContainerStarted","Data":"3b02cf028c881fe72942d0933b3a66de4815b8730343e162ac929b30968f51b2"} Nov 03 01:35:53.874555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:53.874521 5144 generic.go:356] "Generic (PLEG): container finished" podID="c70ac437-31a0-41ac-be02-af0aa2659d62" containerID="79f8ca5b2c56b529e755e5562501065591fc087142bdcd1cf833c88e9bb73448" exitCode=0 Nov 03 01:35:53.874980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:53.874580 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" event={"ID":"c70ac437-31a0-41ac-be02-af0aa2659d62","Type":"ContainerDied","Data":"79f8ca5b2c56b529e755e5562501065591fc087142bdcd1cf833c88e9bb73448"} Nov 03 01:35:55.035447 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.035422 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:55.132986 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.132885 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca\") pod \"c70ac437-31a0-41ac-be02-af0aa2659d62\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " Nov 03 01:35:55.132986 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.132935 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca\") pod \"c70ac437-31a0-41ac-be02-af0aa2659d62\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " Nov 03 01:35:55.133255 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.133012 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km\" (UniqueName: \"kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km\") pod \"c70ac437-31a0-41ac-be02-af0aa2659d62\" (UID: \"c70ac437-31a0-41ac-be02-af0aa2659d62\") " Nov 03 01:35:55.133362 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.133340 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca" pod "c70ac437-31a0-41ac-be02-af0aa2659d62" (UID: "c70ac437-31a0-41ac-be02-af0aa2659d62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:35:55.135420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.135391 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km" pod "c70ac437-31a0-41ac-be02-af0aa2659d62" (UID: "c70ac437-31a0-41ac-be02-af0aa2659d62"). InnerVolumeSpecName "kube-api-access-lr2km". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:35:55.135534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.135437 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca" pod "c70ac437-31a0-41ac-be02-af0aa2659d62" (UID: "c70ac437-31a0-41ac-be02-af0aa2659d62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:35:55.234310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.234267 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:55.234310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.234307 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c70ac437-31a0-41ac-be02-af0aa2659d62-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:55.234310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.234318 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr2km\" (UniqueName: \"kubernetes.io/projected/c70ac437-31a0-41ac-be02-af0aa2659d62-kube-api-access-lr2km\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:35:55.887086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.887054 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" Nov 03 01:35:55.887276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.887093 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq" event={"ID":"c70ac437-31a0-41ac-be02-af0aa2659d62","Type":"ContainerDied","Data":"3b02cf028c881fe72942d0933b3a66de4815b8730343e162ac929b30968f51b2"} Nov 03 01:35:55.887276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:35:55.887142 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b02cf028c881fe72942d0933b3a66de4815b8730343e162ac929b30968f51b2" Nov 03 01:36:04.417748 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.417710 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw"] Nov 03 01:36:04.418347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.418185 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c70ac437-31a0-41ac-be02-af0aa2659d62" containerName="revoke-keys" Nov 03 01:36:04.418347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.418196 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70ac437-31a0-41ac-be02-af0aa2659d62" containerName="revoke-keys" Nov 03 01:36:04.418347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.418269 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70ac437-31a0-41ac-be02-af0aa2659d62" containerName="revoke-keys" Nov 03 01:36:04.421820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.421796 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.438099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.438055 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq62g\" (UniqueName: \"kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.438300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.438175 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.438300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.438221 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.477045 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.477009 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw"] Nov 03 01:36:04.522851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.522815 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:36:04.523087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.522879 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:36:04.538725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.538687 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq62g\" (UniqueName: \"kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.538943 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.538764 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.538943 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.538800 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.539441 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.539418 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.541340 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.541319 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.596975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.596944 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq62g\" (UniqueName: \"kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g\") pod \"maas-api-revoke-sub-ec843018-tr6jw\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:04.736444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:04.736351 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:05.121838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:05.121801 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw"] Nov 03 01:36:05.946285 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:05.946244 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" event={"ID":"0de3af84-8739-49b4-93ef-fbc7b9133af2","Type":"ContainerStarted","Data":"a0a55e71e56d643788c30f951690adeb5c7ac7526774211708a3161c97da9223"} Nov 03 01:36:05.946757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:05.946309 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" event={"ID":"0de3af84-8739-49b4-93ef-fbc7b9133af2","Type":"ContainerStarted","Data":"558dd45309483bb2643bec03e5aa3d04fd67f94250e7c3677290d9810b0dbc59"} Nov 03 01:36:06.058736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:06.058682 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" podStartSLOduration=2.058668514 podStartE2EDuration="2.058668514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:06.05794912 +0000 UTC m=+5456.203643593" watchObservedRunningTime="2026-11-03 01:36:06.058668514 +0000 UTC m=+5456.204362968" Nov 03 01:36:06.954190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:06.954155 5144 generic.go:356] "Generic (PLEG): container finished" podID="0de3af84-8739-49b4-93ef-fbc7b9133af2" containerID="a0a55e71e56d643788c30f951690adeb5c7ac7526774211708a3161c97da9223" exitCode=0 Nov 03 01:36:06.954609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:06.954215 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" event={"ID":"0de3af84-8739-49b4-93ef-fbc7b9133af2","Type":"ContainerDied","Data":"a0a55e71e56d643788c30f951690adeb5c7ac7526774211708a3161c97da9223"} Nov 03 01:36:08.114212 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.114185 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:08.175976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.175942 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca\") pod \"0de3af84-8739-49b4-93ef-fbc7b9133af2\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " Nov 03 01:36:08.175976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.175982 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g\" (UniqueName: \"kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g\") pod \"0de3af84-8739-49b4-93ef-fbc7b9133af2\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " Nov 03 01:36:08.176261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.176045 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca\") pod \"0de3af84-8739-49b4-93ef-fbc7b9133af2\" (UID: \"0de3af84-8739-49b4-93ef-fbc7b9133af2\") " Nov 03 01:36:08.176492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.176465 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca" pod "0de3af84-8739-49b4-93ef-fbc7b9133af2" (UID: "0de3af84-8739-49b4-93ef-fbc7b9133af2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:08.178390 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.178366 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca" pod "0de3af84-8739-49b4-93ef-fbc7b9133af2" (UID: "0de3af84-8739-49b4-93ef-fbc7b9133af2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:08.178390 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.178376 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g" pod "0de3af84-8739-49b4-93ef-fbc7b9133af2" (UID: "0de3af84-8739-49b4-93ef-fbc7b9133af2"). InnerVolumeSpecName "kube-api-access-sq62g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:08.276997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.276959 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:08.276997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.276992 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq62g\" (UniqueName: \"kubernetes.io/projected/0de3af84-8739-49b4-93ef-fbc7b9133af2-kube-api-access-sq62g\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:08.276997 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.277002 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0de3af84-8739-49b4-93ef-fbc7b9133af2-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:08.967910 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.967879 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" Nov 03 01:36:08.968134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.967926 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw" event={"ID":"0de3af84-8739-49b4-93ef-fbc7b9133af2","Type":"ContainerDied","Data":"558dd45309483bb2643bec03e5aa3d04fd67f94250e7c3677290d9810b0dbc59"} Nov 03 01:36:08.968134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:08.967983 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="558dd45309483bb2643bec03e5aa3d04fd67f94250e7c3677290d9810b0dbc59" Nov 03 01:36:12.155018 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:36:12.154978 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:36:14.780356 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:14.780324 5144 scope.go:122] "RemoveContainer" containerID="52caf10591cf88aaf9bad8d39e3f2c47274d7a335d83063e85697fbf7d727bf4" Nov 03 01:36:28.044183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:28.044147 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf"] Nov 03 01:36:28.093952 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:28.093918 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-bvfzf"] Nov 03 01:36:29.995682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:29.995613 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc5ff267-22df-467c-a2a4-068c10e9bc2e" path="/var/lib/kubelet/pods/cc5ff267-22df-467c-a2a4-068c10e9bc2e/volumes" Nov 03 01:36:36.507886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.507849 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7"] Nov 03 01:36:36.508398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.508301 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0de3af84-8739-49b4-93ef-fbc7b9133af2" containerName="revoke-keys" Nov 03 01:36:36.508398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.508311 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0de3af84-8739-49b4-93ef-fbc7b9133af2" containerName="revoke-keys" Nov 03 01:36:36.508398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.508388 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0de3af84-8739-49b4-93ef-fbc7b9133af2" containerName="revoke-keys" Nov 03 01:36:36.512079 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.512063 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.569519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.569469 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7"] Nov 03 01:36:36.605316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.605251 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.605547 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.605372 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9gv6\" (UniqueName: \"kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.605547 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.605410 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.615228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.615196 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:36:36.615387 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.615256 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:36:36.706602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.706561 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.706856 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.706618 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9gv6\" (UniqueName: \"kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.706856 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.706676 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.707454 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.707384 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.709372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.709348 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.763296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.763208 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9gv6\" (UniqueName: \"kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6\") pod \"maas-api-revoke-sub-046e1343-pc4b7\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:36.825263 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:36.825224 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:37.197925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:37.197899 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7"] Nov 03 01:36:37.200121 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:36:37.200089 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdfceec_9aa6_4257_9f7d_f7b90a827641.slice/crio-9ef0f1cb546f7aed269ae738fbdc4a1734d030015ed5af9c73e4e5cbd187a614 WatchSource:0}: Error finding container 9ef0f1cb546f7aed269ae738fbdc4a1734d030015ed5af9c73e4e5cbd187a614: Status 404 returned error can't find the container with id 9ef0f1cb546f7aed269ae738fbdc4a1734d030015ed5af9c73e4e5cbd187a614 Nov 03 01:36:38.162694 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:38.162624 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" event={"ID":"6fdfceec-9aa6-4257-9f7d-f7b90a827641","Type":"ContainerStarted","Data":"648822f980af55f448c9b4b33bb42dbc4500b69194e3ce518a939934db508b33"} Nov 03 01:36:38.163116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:38.162715 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" event={"ID":"6fdfceec-9aa6-4257-9f7d-f7b90a827641","Type":"ContainerStarted","Data":"9ef0f1cb546f7aed269ae738fbdc4a1734d030015ed5af9c73e4e5cbd187a614"} Nov 03 01:36:38.276364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:38.276299 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" podStartSLOduration=2.276284047 podStartE2EDuration="2.276284047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:38.274371822 +0000 UTC m=+5488.420066279" watchObservedRunningTime="2026-11-03 01:36:38.276284047 +0000 UTC m=+5488.421978502" Nov 03 01:36:39.169280 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:39.169247 5144 generic.go:356] "Generic (PLEG): container finished" podID="6fdfceec-9aa6-4257-9f7d-f7b90a827641" containerID="648822f980af55f448c9b4b33bb42dbc4500b69194e3ce518a939934db508b33" exitCode=0 Nov 03 01:36:39.169760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:39.169321 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" event={"ID":"6fdfceec-9aa6-4257-9f7d-f7b90a827641","Type":"ContainerDied","Data":"648822f980af55f448c9b4b33bb42dbc4500b69194e3ce518a939934db508b33"} Nov 03 01:36:40.324473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.324441 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:40.349922 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.349888 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca\") pod \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " Nov 03 01:36:40.350078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.349925 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6\" (UniqueName: \"kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6\") pod \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " Nov 03 01:36:40.350078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.349974 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca\") pod \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\" (UID: \"6fdfceec-9aa6-4257-9f7d-f7b90a827641\") " Nov 03 01:36:40.350381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.350349 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca" pod "6fdfceec-9aa6-4257-9f7d-f7b90a827641" (UID: "6fdfceec-9aa6-4257-9f7d-f7b90a827641"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:40.350502 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.350477 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:40.352243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.352217 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca" pod "6fdfceec-9aa6-4257-9f7d-f7b90a827641" (UID: "6fdfceec-9aa6-4257-9f7d-f7b90a827641"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:40.352350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.352242 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6" pod "6fdfceec-9aa6-4257-9f7d-f7b90a827641" (UID: "6fdfceec-9aa6-4257-9f7d-f7b90a827641"). InnerVolumeSpecName "kube-api-access-l9gv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:40.451782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.451706 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fdfceec-9aa6-4257-9f7d-f7b90a827641-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:40.451782 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:40.451734 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9gv6\" (UniqueName: \"kubernetes.io/projected/6fdfceec-9aa6-4257-9f7d-f7b90a827641-kube-api-access-l9gv6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:41.182700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:41.182666 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" Nov 03 01:36:41.182700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:41.182688 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7" event={"ID":"6fdfceec-9aa6-4257-9f7d-f7b90a827641","Type":"ContainerDied","Data":"9ef0f1cb546f7aed269ae738fbdc4a1734d030015ed5af9c73e4e5cbd187a614"} Nov 03 01:36:41.182928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:41.182739 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ef0f1cb546f7aed269ae738fbdc4a1734d030015ed5af9c73e4e5cbd187a614" Nov 03 01:36:42.154858 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:36:42.154813 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:36:47.267417 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.267368 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf"] Nov 03 01:36:47.267969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.267851 5144 cpu_manager.go:414] "containerMap: removing container" podUID="6fdfceec-9aa6-4257-9f7d-f7b90a827641" containerName="revoke-keys" Nov 03 01:36:47.267969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.267863 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fdfceec-9aa6-4257-9f7d-f7b90a827641" containerName="revoke-keys" Nov 03 01:36:47.267969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.267925 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fdfceec-9aa6-4257-9f7d-f7b90a827641" containerName="revoke-keys" Nov 03 01:36:47.271284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.271263 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.324376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.324331 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.324676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.324647 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.324813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.324745 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmh9q\" (UniqueName: \"kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.327075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.327028 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf"] Nov 03 01:36:47.372931 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.372895 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:36:47.373158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.372990 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:36:47.425526 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.425492 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.425526 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.425534 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.425823 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.425584 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmh9q\" (UniqueName: \"kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.426256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.426232 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.428202 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.428170 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.482343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.482296 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmh9q\" (UniqueName: \"kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q\") pod \"maas-api-revoke-sub-d3fba056-r9flf\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.586025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.585934 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:47.973246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:47.973221 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf"] Nov 03 01:36:47.974855 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:36:47.974828 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8278fe4_cfbb_405a_b32f_fe9afbd83eda.slice/crio-d55f252fe6c3f779e053eea08048a67c6259724f42d5496ea81af94bcb47f46e WatchSource:0}: Error finding container d55f252fe6c3f779e053eea08048a67c6259724f42d5496ea81af94bcb47f46e: Status 404 returned error can't find the container with id d55f252fe6c3f779e053eea08048a67c6259724f42d5496ea81af94bcb47f46e Nov 03 01:36:48.225062 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:48.225030 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" event={"ID":"f8278fe4-cfbb-405a-b32f-fe9afbd83eda","Type":"ContainerStarted","Data":"d55f252fe6c3f779e053eea08048a67c6259724f42d5496ea81af94bcb47f46e"} Nov 03 01:36:49.232528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:49.232434 5144 generic.go:356] "Generic (PLEG): container finished" podID="f8278fe4-cfbb-405a-b32f-fe9afbd83eda" containerID="66b87ab02ad1ce2437e3553f8a599d1f0ec3f5b21e0433bd2454f606e57ebd43" exitCode=0 Nov 03 01:36:49.232958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:49.232529 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" event={"ID":"f8278fe4-cfbb-405a-b32f-fe9afbd83eda","Type":"ContainerDied","Data":"66b87ab02ad1ce2437e3553f8a599d1f0ec3f5b21e0433bd2454f606e57ebd43"} Nov 03 01:36:50.438815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.438776 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:50.562897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.561292 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca\") pod \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " Nov 03 01:36:50.562897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.561505 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q\" (UniqueName: \"kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q\") pod \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " Nov 03 01:36:50.562897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.561563 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca\") pod \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\" (UID: \"f8278fe4-cfbb-405a-b32f-fe9afbd83eda\") " Nov 03 01:36:50.562897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.562585 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca" pod "f8278fe4-cfbb-405a-b32f-fe9afbd83eda" (UID: "f8278fe4-cfbb-405a-b32f-fe9afbd83eda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:36:50.580655 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.580583 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca" pod "f8278fe4-cfbb-405a-b32f-fe9afbd83eda" (UID: "f8278fe4-cfbb-405a-b32f-fe9afbd83eda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:36:50.583864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.583818 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q" pod "f8278fe4-cfbb-405a-b32f-fe9afbd83eda" (UID: "f8278fe4-cfbb-405a-b32f-fe9afbd83eda"). InnerVolumeSpecName "kube-api-access-zmh9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:36:50.673868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.673759 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmh9q\" (UniqueName: \"kubernetes.io/projected/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-kube-api-access-zmh9q\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:50.673868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.673801 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:50.673868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:50.673830 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8278fe4-cfbb-405a-b32f-fe9afbd83eda-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:36:51.260656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:51.259735 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" event={"ID":"f8278fe4-cfbb-405a-b32f-fe9afbd83eda","Type":"ContainerDied","Data":"d55f252fe6c3f779e053eea08048a67c6259724f42d5496ea81af94bcb47f46e"} Nov 03 01:36:51.260656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:51.259803 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d55f252fe6c3f779e053eea08048a67c6259724f42d5496ea81af94bcb47f46e" Nov 03 01:36:51.260656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:51.259872 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf" Nov 03 01:36:51.997864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:51.997819 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:36:51.998420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:51.997889 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw" podUID="43f74203-a9e7-4cfe-ab2a-596a0ea37e1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:36:54.045717 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:54.045681 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh"] Nov 03 01:36:54.095559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:54.095526 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-v5wlh"] Nov 03 01:36:55.994463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:36:55.994430 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc592391-fc11-4097-86b4-2b7e9290ccb7" path="/var/lib/kubelet/pods/bc592391-fc11-4097-86b4-2b7e9290ccb7/volumes" Nov 03 01:37:12.156573 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:37:12.156523 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:37:18.044686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:18.044649 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd"] Nov 03 01:37:18.093797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:18.093758 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e15a4e09-jmfzd"] Nov 03 01:37:19.995599 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:19.995520 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37b5a70-18cd-4802-9f40-1f250c350006" path="/var/lib/kubelet/pods/c37b5a70-18cd-4802-9f40-1f250c350006/volumes" Nov 03 01:37:21.917180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:21.917144 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st"] Nov 03 01:37:21.917667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:21.917623 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f8278fe4-cfbb-405a-b32f-fe9afbd83eda" containerName="revoke-keys" Nov 03 01:37:21.917667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:21.917652 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8278fe4-cfbb-405a-b32f-fe9afbd83eda" containerName="revoke-keys" Nov 03 01:37:21.917767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:21.917757 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8278fe4-cfbb-405a-b32f-fe9afbd83eda" containerName="revoke-keys" Nov 03 01:37:21.922406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:21.922381 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:21.975799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:21.975767 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st"] Nov 03 01:37:22.022452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.022413 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:37:22.022683 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.022466 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:37:22.023531 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.023510 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpl9s\" (UniqueName: \"kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.023649 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.023623 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.023699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.023682 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.124502 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.124462 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.124767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.124518 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.124767 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.124572 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpl9s\" (UniqueName: \"kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.125224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.125198 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.127126 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.127107 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.181282 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.181195 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpl9s\" (UniqueName: \"kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s\") pod \"maas-api-revoke-sub-ae594979-n29st\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.236470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.236434 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:22.607514 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:22.607485 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st"] Nov 03 01:37:22.609347 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:37:22.609322 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a59b80a_3ff2_4348_b248_0304f9cd5189.slice/crio-98f8699ec628b9a6168309d57f970ae6261b4b9eb559dff9fe727576e67b5ecd WatchSource:0}: Error finding container 98f8699ec628b9a6168309d57f970ae6261b4b9eb559dff9fe727576e67b5ecd: Status 404 returned error can't find the container with id 98f8699ec628b9a6168309d57f970ae6261b4b9eb559dff9fe727576e67b5ecd Nov 03 01:37:23.489780 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:23.489746 5144 generic.go:356] "Generic (PLEG): container finished" podID="4a59b80a-3ff2-4348-b248-0304f9cd5189" containerID="a080127e8b3ac77dda3360764f68081e601a263d3ecda224463144d08c05d671" exitCode=0 Nov 03 01:37:23.490157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:23.489828 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" event={"ID":"4a59b80a-3ff2-4348-b248-0304f9cd5189","Type":"ContainerDied","Data":"a080127e8b3ac77dda3360764f68081e601a263d3ecda224463144d08c05d671"} Nov 03 01:37:23.490157 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:23.489898 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" event={"ID":"4a59b80a-3ff2-4348-b248-0304f9cd5189","Type":"ContainerStarted","Data":"98f8699ec628b9a6168309d57f970ae6261b4b9eb559dff9fe727576e67b5ecd"} Nov 03 01:37:24.636825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.636798 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:24.749590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.749503 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca\") pod \"4a59b80a-3ff2-4348-b248-0304f9cd5189\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " Nov 03 01:37:24.749802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.749611 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s\" (UniqueName: \"kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s\") pod \"4a59b80a-3ff2-4348-b248-0304f9cd5189\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " Nov 03 01:37:24.749802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.749708 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca\") pod \"4a59b80a-3ff2-4348-b248-0304f9cd5189\" (UID: \"4a59b80a-3ff2-4348-b248-0304f9cd5189\") " Nov 03 01:37:24.750354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.750312 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca" pod "4a59b80a-3ff2-4348-b248-0304f9cd5189" (UID: "4a59b80a-3ff2-4348-b248-0304f9cd5189"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:24.752472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.752445 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca" pod "4a59b80a-3ff2-4348-b248-0304f9cd5189" (UID: "4a59b80a-3ff2-4348-b248-0304f9cd5189"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:37:24.752695 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.752671 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s" pod "4a59b80a-3ff2-4348-b248-0304f9cd5189" (UID: "4a59b80a-3ff2-4348-b248-0304f9cd5189"). InnerVolumeSpecName "kube-api-access-zpl9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:24.851464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.851429 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:37:24.851464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.851459 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpl9s\" (UniqueName: \"kubernetes.io/projected/4a59b80a-3ff2-4348-b248-0304f9cd5189-kube-api-access-zpl9s\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:37:24.851464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:24.851471 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a59b80a-3ff2-4348-b248-0304f9cd5189-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:37:25.502671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:25.502619 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" event={"ID":"4a59b80a-3ff2-4348-b248-0304f9cd5189","Type":"ContainerDied","Data":"98f8699ec628b9a6168309d57f970ae6261b4b9eb559dff9fe727576e67b5ecd"} Nov 03 01:37:25.502671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:25.502659 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st" Nov 03 01:37:25.502904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:25.502685 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f8699ec628b9a6168309d57f970ae6261b4b9eb559dff9fe727576e67b5ecd" Nov 03 01:37:42.220658 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:37:42.218501 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:37:47.457116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.457073 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd"] Nov 03 01:37:47.457944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.457538 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4a59b80a-3ff2-4348-b248-0304f9cd5189" containerName="revoke-keys" Nov 03 01:37:47.457944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.457549 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a59b80a-3ff2-4348-b248-0304f9cd5189" containerName="revoke-keys" Nov 03 01:37:47.457944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.457617 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a59b80a-3ff2-4348-b248-0304f9cd5189" containerName="revoke-keys" Nov 03 01:37:47.460916 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.460899 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.513565 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.513517 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:37:47.516941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.516908 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd"] Nov 03 01:37:47.562394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.562348 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:37:47.626384 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.626343 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.626584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.626526 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.626584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.626565 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.727686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.727568 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.727865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.727715 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.727865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.727744 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.728418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.728391 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.730330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.730306 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:47.784986 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:47.784955 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848\") pod \"maas-api-revoke-sub-a4aef7d7-cvcfd\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:48.075104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:48.075061 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:48.452583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:48.452549 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd"] Nov 03 01:37:48.454350 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:37:48.454316 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efb5d90_e3a5_4597_be78_ef818d3ade4c.slice/crio-e74f489719cb8135b237b8cf4193f4f98826791ec1338baa7152aa56f17a4e6a WatchSource:0}: Error finding container e74f489719cb8135b237b8cf4193f4f98826791ec1338baa7152aa56f17a4e6a: Status 404 returned error can't find the container with id e74f489719cb8135b237b8cf4193f4f98826791ec1338baa7152aa56f17a4e6a Nov 03 01:37:48.642619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:48.642587 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" event={"ID":"8efb5d90-e3a5-4597-be78-ef818d3ade4c","Type":"ContainerStarted","Data":"e74f489719cb8135b237b8cf4193f4f98826791ec1338baa7152aa56f17a4e6a"} Nov 03 01:37:49.650191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:49.650088 5144 generic.go:356] "Generic (PLEG): container finished" podID="8efb5d90-e3a5-4597-be78-ef818d3ade4c" containerID="8cc7518a327221b3d1fd53e6c941b52df16d8b27ea466f00c8061fbf8e29b57d" exitCode=0 Nov 03 01:37:49.650191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:49.650175 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" event={"ID":"8efb5d90-e3a5-4597-be78-ef818d3ade4c","Type":"ContainerDied","Data":"8cc7518a327221b3d1fd53e6c941b52df16d8b27ea466f00c8061fbf8e29b57d"} Nov 03 01:37:50.813240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.813210 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:50.960905 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.960795 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca\") pod \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " Nov 03 01:37:50.961101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.960961 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca\") pod \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " Nov 03 01:37:50.961101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.961007 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848\") pod \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\" (UID: \"8efb5d90-e3a5-4597-be78-ef818d3ade4c\") " Nov 03 01:37:50.961666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.961602 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca" pod "8efb5d90-e3a5-4597-be78-ef818d3ade4c" (UID: "8efb5d90-e3a5-4597-be78-ef818d3ade4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:37:50.963758 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.963730 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca" pod "8efb5d90-e3a5-4597-be78-ef818d3ade4c" (UID: "8efb5d90-e3a5-4597-be78-ef818d3ade4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:37:50.963876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:50.963773 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848" pod "8efb5d90-e3a5-4597-be78-ef818d3ade4c" (UID: "8efb5d90-e3a5-4597-be78-ef818d3ade4c"). InnerVolumeSpecName "kube-api-access-2q848". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:37:51.062218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:51.062180 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:37:51.062218 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:51.062215 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q848\" (UniqueName: \"kubernetes.io/projected/8efb5d90-e3a5-4597-be78-ef818d3ade4c-kube-api-access-2q848\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:37:51.062455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:51.062228 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8efb5d90-e3a5-4597-be78-ef818d3ade4c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:37:51.664043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:51.664007 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" event={"ID":"8efb5d90-e3a5-4597-be78-ef818d3ade4c","Type":"ContainerDied","Data":"e74f489719cb8135b237b8cf4193f4f98826791ec1338baa7152aa56f17a4e6a"} Nov 03 01:37:51.664043 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:51.664026 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd" Nov 03 01:37:51.664285 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:37:51.664063 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e74f489719cb8135b237b8cf4193f4f98826791ec1338baa7152aa56f17a4e6a" Nov 03 01:38:08.514916 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.514878 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt"] Nov 03 01:38:08.516197 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.516171 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8efb5d90-e3a5-4597-be78-ef818d3ade4c" containerName="revoke-keys" Nov 03 01:38:08.516377 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.516361 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8efb5d90-e3a5-4597-be78-ef818d3ade4c" containerName="revoke-keys" Nov 03 01:38:08.516699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.516680 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8efb5d90-e3a5-4597-be78-ef818d3ade4c" containerName="revoke-keys" Nov 03 01:38:08.526979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.526944 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.596657 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.595899 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt"] Nov 03 01:38:08.660983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.660950 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:38:08.661275 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.661258 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:38:08.766973 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.766879 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.767281 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.767253 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mjmk\" (UniqueName: \"kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.767429 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.767413 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.868690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.868624 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mjmk\" (UniqueName: \"kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.868690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.868688 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.868940 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.868861 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.869543 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.869517 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.871525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.871489 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:08.926259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:08.926225 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mjmk\" (UniqueName: \"kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk\") pod \"maas-api-revoke-sub-79805ca9-zfjgt\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:09.164192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:09.164101 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:09.535715 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:09.535688 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt"] Nov 03 01:38:09.537159 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:38:09.537132 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73c1934e_91bb_45dc_9683_1ed5de03c6a5.slice/crio-7c7944a05b3ad09348666c967c4ecae44e03e3ebabdf8b0ce708894517232080 WatchSource:0}: Error finding container 7c7944a05b3ad09348666c967c4ecae44e03e3ebabdf8b0ce708894517232080: Status 404 returned error can't find the container with id 7c7944a05b3ad09348666c967c4ecae44e03e3ebabdf8b0ce708894517232080 Nov 03 01:38:09.815276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:09.815187 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" event={"ID":"73c1934e-91bb-45dc-9683-1ed5de03c6a5","Type":"ContainerStarted","Data":"7c7944a05b3ad09348666c967c4ecae44e03e3ebabdf8b0ce708894517232080"} Nov 03 01:38:10.823240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:10.823200 5144 generic.go:356] "Generic (PLEG): container finished" podID="73c1934e-91bb-45dc-9683-1ed5de03c6a5" containerID="cfeb1975bf6107e55c34b5638f80342f4ac0c8339405d8dc76bbae261a987b22" exitCode=0 Nov 03 01:38:10.823690 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:10.823285 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" event={"ID":"73c1934e-91bb-45dc-9683-1ed5de03c6a5","Type":"ContainerDied","Data":"cfeb1975bf6107e55c34b5638f80342f4ac0c8339405d8dc76bbae261a987b22"} Nov 03 01:38:11.970333 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:11.970303 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:12.000969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.000938 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca\") pod \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " Nov 03 01:38:12.001175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.000979 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca\") pod \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " Nov 03 01:38:12.001175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.001021 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk\" (UniqueName: \"kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk\") pod \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\" (UID: \"73c1934e-91bb-45dc-9683-1ed5de03c6a5\") " Nov 03 01:38:12.001350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.001326 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca" pod "73c1934e-91bb-45dc-9683-1ed5de03c6a5" (UID: "73c1934e-91bb-45dc-9683-1ed5de03c6a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:12.003251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.003222 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk" pod "73c1934e-91bb-45dc-9683-1ed5de03c6a5" (UID: "73c1934e-91bb-45dc-9683-1ed5de03c6a5"). InnerVolumeSpecName "kube-api-access-4mjmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:12.003347 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.003232 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca" pod "73c1934e-91bb-45dc-9683-1ed5de03c6a5" (UID: "73c1934e-91bb-45dc-9683-1ed5de03c6a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:12.102189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.102101 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:12.102189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.102135 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73c1934e-91bb-45dc-9683-1ed5de03c6a5-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:12.102189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.102145 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mjmk\" (UniqueName: \"kubernetes.io/projected/73c1934e-91bb-45dc-9683-1ed5de03c6a5-kube-api-access-4mjmk\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:12.155610 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:38:12.155563 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:38:12.836432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.836405 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" Nov 03 01:38:12.836617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.836403 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt" event={"ID":"73c1934e-91bb-45dc-9683-1ed5de03c6a5","Type":"ContainerDied","Data":"7c7944a05b3ad09348666c967c4ecae44e03e3ebabdf8b0ce708894517232080"} Nov 03 01:38:12.836617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:12.836524 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7944a05b3ad09348666c967c4ecae44e03e3ebabdf8b0ce708894517232080" Nov 03 01:38:14.814487 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:14.814446 5144 scope.go:122] "RemoveContainer" containerID="aadf5c86e943eb9bc696fea4c8a079b4244767d2d8e88dcbcf43b593f59b2dc9" Nov 03 01:38:14.824234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:14.824207 5144 scope.go:122] "RemoveContainer" containerID="ff2452722b99a83dacf5dbdd5f869e44d7e3811fd023cb58f179ab42b39235c7" Nov 03 01:38:14.833196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:14.833173 5144 scope.go:122] "RemoveContainer" containerID="62448ddf8c272fb263b4093e5a542d20f9f74639a3a77b10368775e914651c91" Nov 03 01:38:26.033864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.033825 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5"] Nov 03 01:38:26.035337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.035301 5144 cpu_manager.go:414] "containerMap: removing container" podUID="73c1934e-91bb-45dc-9683-1ed5de03c6a5" containerName="revoke-keys" Nov 03 01:38:26.035524 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.035509 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c1934e-91bb-45dc-9683-1ed5de03c6a5" containerName="revoke-keys" Nov 03 01:38:26.035897 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.035881 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c1934e-91bb-45dc-9683-1ed5de03c6a5" containerName="revoke-keys" Nov 03 01:38:26.050070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.050041 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.107741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.107693 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5"] Nov 03 01:38:26.154044 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.154014 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:38:26.154428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.154399 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:38:26.207619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.207329 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.207619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.207416 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.207619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.207443 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bztc8\" (UniqueName: \"kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.308320 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.308237 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.308560 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.308544 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.308682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.308664 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bztc8\" (UniqueName: \"kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.309077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.309047 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.311116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.311089 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.371656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.371358 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bztc8\" (UniqueName: \"kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8\") pod \"maas-api-revoke-sub-0b7e8edb-vdkb5\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.423885 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.423845 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:26.932243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:26.932210 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5"] Nov 03 01:38:26.934130 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:38:26.934090 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b9caccb_6d8f_4b51_9cf3_759e9e0f0b18.slice/crio-5f9eba2233765fcf3034959e914c23875ed4d14f5854c1aa85517256a053a6c4 WatchSource:0}: Error finding container 5f9eba2233765fcf3034959e914c23875ed4d14f5854c1aa85517256a053a6c4: Status 404 returned error can't find the container with id 5f9eba2233765fcf3034959e914c23875ed4d14f5854c1aa85517256a053a6c4 Nov 03 01:38:27.942208 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:27.942166 5144 generic.go:356] "Generic (PLEG): container finished" podID="8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" containerID="2d4e4b2e3f97446d04b86b78fbfca3552f13aadc5dd42904f2b8bdaa528e489d" exitCode=0 Nov 03 01:38:27.942742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:27.942253 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" event={"ID":"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18","Type":"ContainerDied","Data":"2d4e4b2e3f97446d04b86b78fbfca3552f13aadc5dd42904f2b8bdaa528e489d"} Nov 03 01:38:27.942742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:27.942332 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" event={"ID":"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18","Type":"ContainerStarted","Data":"5f9eba2233765fcf3034959e914c23875ed4d14f5854c1aa85517256a053a6c4"} Nov 03 01:38:29.089475 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.089448 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:29.139716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.139678 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca\") pod \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " Nov 03 01:38:29.139890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.139808 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca\") pod \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " Nov 03 01:38:29.140041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.140017 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca" pod "8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" (UID: "8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:29.142027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.142005 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca" pod "8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" (UID: "8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:29.240784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.240696 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8\" (UniqueName: \"kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8\") pod \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\" (UID: \"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18\") " Nov 03 01:38:29.240960 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.240946 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:29.241003 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.240963 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:29.242937 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.242911 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8" pod "8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" (UID: "8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18"). InnerVolumeSpecName "kube-api-access-bztc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:29.341885 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.341842 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bztc8\" (UniqueName: \"kubernetes.io/projected/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18-kube-api-access-bztc8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:29.955442 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.955412 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" Nov 03 01:38:29.955642 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.955456 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5" event={"ID":"8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18","Type":"ContainerDied","Data":"5f9eba2233765fcf3034959e914c23875ed4d14f5854c1aa85517256a053a6c4"} Nov 03 01:38:29.955642 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:29.955507 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f9eba2233765fcf3034959e914c23875ed4d14f5854c1aa85517256a053a6c4" Nov 03 01:38:37.290579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.290531 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p"] Nov 03 01:38:37.293231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.290995 5144 cpu_manager.go:414] "containerMap: removing container" podUID="8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" containerName="revoke-keys" Nov 03 01:38:37.293231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.291006 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" containerName="revoke-keys" Nov 03 01:38:37.293231 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.291105 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" containerName="revoke-keys" Nov 03 01:38:37.294225 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.294208 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.316563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.316531 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrlp6\" (UniqueName: \"kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.316774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.316680 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.316774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.316721 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.395839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.395798 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:38:37.395839 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.395833 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:38:37.398875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.398846 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p"] Nov 03 01:38:37.418223 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.418191 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrlp6\" (UniqueName: \"kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.418418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.418267 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.418418 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.418286 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.419009 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.418990 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.420822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.420807 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.474457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.474425 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrlp6\" (UniqueName: \"kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6\") pod \"maas-api-revoke-sub-3d4156f1-dc82p\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.608888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.608799 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:37.981883 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:37.981855 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p"] Nov 03 01:38:37.982721 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:38:37.982698 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c13d64_2090_4afb_88c0_7ed0ed15419f.slice/crio-21b3b7954f20f5d2ea1c371198e9ee25e1fefecd7fcd5fe5db4b8514734361f3 WatchSource:0}: Error finding container 21b3b7954f20f5d2ea1c371198e9ee25e1fefecd7fcd5fe5db4b8514734361f3: Status 404 returned error can't find the container with id 21b3b7954f20f5d2ea1c371198e9ee25e1fefecd7fcd5fe5db4b8514734361f3 Nov 03 01:38:38.002820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:38.002791 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" event={"ID":"44c13d64-2090-4afb-88c0-7ed0ed15419f","Type":"ContainerStarted","Data":"21b3b7954f20f5d2ea1c371198e9ee25e1fefecd7fcd5fe5db4b8514734361f3"} Nov 03 01:38:39.010190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:39.010156 5144 generic.go:356] "Generic (PLEG): container finished" podID="44c13d64-2090-4afb-88c0-7ed0ed15419f" containerID="bfb4d21d56f09323e9c066ae41418562d1d331d090aad54ffb60166e5946142b" exitCode=0 Nov 03 01:38:39.010606 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:39.010238 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" event={"ID":"44c13d64-2090-4afb-88c0-7ed0ed15419f","Type":"ContainerDied","Data":"bfb4d21d56f09323e9c066ae41418562d1d331d090aad54ffb60166e5946142b"} Nov 03 01:38:40.158909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.158884 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:40.249962 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.249926 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca\") pod \"44c13d64-2090-4afb-88c0-7ed0ed15419f\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " Nov 03 01:38:40.250147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.250069 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca\") pod \"44c13d64-2090-4afb-88c0-7ed0ed15419f\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " Nov 03 01:38:40.250147 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.250114 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6\" (UniqueName: \"kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6\") pod \"44c13d64-2090-4afb-88c0-7ed0ed15419f\" (UID: \"44c13d64-2090-4afb-88c0-7ed0ed15419f\") " Nov 03 01:38:40.250545 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.250512 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca" pod "44c13d64-2090-4afb-88c0-7ed0ed15419f" (UID: "44c13d64-2090-4afb-88c0-7ed0ed15419f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:38:40.252260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.252235 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca" pod "44c13d64-2090-4afb-88c0-7ed0ed15419f" (UID: "44c13d64-2090-4afb-88c0-7ed0ed15419f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:38:40.252403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.252387 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6" pod "44c13d64-2090-4afb-88c0-7ed0ed15419f" (UID: "44c13d64-2090-4afb-88c0-7ed0ed15419f"). InnerVolumeSpecName "kube-api-access-hrlp6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:38:40.351812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.351726 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:40.351812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.351755 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrlp6\" (UniqueName: \"kubernetes.io/projected/44c13d64-2090-4afb-88c0-7ed0ed15419f-kube-api-access-hrlp6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:40.351812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:40.351765 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c13d64-2090-4afb-88c0-7ed0ed15419f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:38:41.023438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:41.023412 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" Nov 03 01:38:41.023651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:41.023404 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p" event={"ID":"44c13d64-2090-4afb-88c0-7ed0ed15419f","Type":"ContainerDied","Data":"21b3b7954f20f5d2ea1c371198e9ee25e1fefecd7fcd5fe5db4b8514734361f3"} Nov 03 01:38:41.023651 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:38:41.023502 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b3b7954f20f5d2ea1c371198e9ee25e1fefecd7fcd5fe5db4b8514734361f3" Nov 03 01:38:42.155528 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:38:42.155488 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:39:01.131267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.131231 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p"] Nov 03 01:39:01.180446 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.180412 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-dc82p"] Nov 03 01:39:01.251646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.251598 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb"] Nov 03 01:39:01.252077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.252063 5144 cpu_manager.go:414] "containerMap: removing container" podUID="44c13d64-2090-4afb-88c0-7ed0ed15419f" containerName="revoke-keys" Nov 03 01:39:01.252121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.252079 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c13d64-2090-4afb-88c0-7ed0ed15419f" containerName="revoke-keys" Nov 03 01:39:01.252179 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.252170 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c13d64-2090-4afb-88c0-7ed0ed15419f" containerName="revoke-keys" Nov 03 01:39:01.255508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.255491 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.308076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.308044 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:39:01.308329 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.308099 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:39:01.311682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.311616 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb"] Nov 03 01:39:01.406822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.406784 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.406822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.406825 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.407053 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.406930 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql26b\" (UniqueName: \"kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.508007 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.507961 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.508007 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.508008 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.508300 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.508057 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql26b\" (UniqueName: \"kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.508799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.508771 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.510702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.510682 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.564905 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.564825 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql26b\" (UniqueName: \"kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b\") pod \"maas-api-revoke-sub-3d4156f1-zptgb\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.568481 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.568458 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:01.940924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.940892 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb"] Nov 03 01:39:01.942491 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:39:01.942458 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a51d5af_f8c6_4df0_a909_f2f663c423c9.slice/crio-0b8ae379007611e7ec918ce7d602bd0090aa469a7d1d5ac396f91047bd4e1ccd WatchSource:0}: Error finding container 0b8ae379007611e7ec918ce7d602bd0090aa469a7d1d5ac396f91047bd4e1ccd: Status 404 returned error can't find the container with id 0b8ae379007611e7ec918ce7d602bd0090aa469a7d1d5ac396f91047bd4e1ccd Nov 03 01:39:01.995105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:01.995071 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c13d64-2090-4afb-88c0-7ed0ed15419f" path="/var/lib/kubelet/pods/44c13d64-2090-4afb-88c0-7ed0ed15419f/volumes" Nov 03 01:39:02.195466 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:02.195374 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" event={"ID":"2a51d5af-f8c6-4df0-a909-f2f663c423c9","Type":"ContainerStarted","Data":"0b8ae379007611e7ec918ce7d602bd0090aa469a7d1d5ac396f91047bd4e1ccd"} Nov 03 01:39:03.204140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:03.204054 5144 generic.go:356] "Generic (PLEG): container finished" podID="2a51d5af-f8c6-4df0-a909-f2f663c423c9" containerID="513b6e6a5287af4ab3302dd9b59fcface9d87fff0ed2cea7e3238e0ceb0e0196" exitCode=0 Nov 03 01:39:03.204140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:03.204107 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" event={"ID":"2a51d5af-f8c6-4df0-a909-f2f663c423c9","Type":"ContainerDied","Data":"513b6e6a5287af4ab3302dd9b59fcface9d87fff0ed2cea7e3238e0ceb0e0196"} Nov 03 01:39:04.345247 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.345223 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:04.438443 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.438406 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca\") pod \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " Nov 03 01:39:04.438696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.438472 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca\") pod \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " Nov 03 01:39:04.438696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.438514 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b\" (UniqueName: \"kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b\") pod \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\" (UID: \"2a51d5af-f8c6-4df0-a909-f2f663c423c9\") " Nov 03 01:39:04.438890 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.438861 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca" pod "2a51d5af-f8c6-4df0-a909-f2f663c423c9" (UID: "2a51d5af-f8c6-4df0-a909-f2f663c423c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:04.440830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.440798 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca" pod "2a51d5af-f8c6-4df0-a909-f2f663c423c9" (UID: "2a51d5af-f8c6-4df0-a909-f2f663c423c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:04.440930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.440894 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b" pod "2a51d5af-f8c6-4df0-a909-f2f663c423c9" (UID: "2a51d5af-f8c6-4df0-a909-f2f663c423c9"). InnerVolumeSpecName "kube-api-access-ql26b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:04.540135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.540092 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:04.540135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.540129 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a51d5af-f8c6-4df0-a909-f2f663c423c9-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:04.540135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:04.540140 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql26b\" (UniqueName: \"kubernetes.io/projected/2a51d5af-f8c6-4df0-a909-f2f663c423c9-kube-api-access-ql26b\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:05.217888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:05.217847 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" event={"ID":"2a51d5af-f8c6-4df0-a909-f2f663c423c9","Type":"ContainerDied","Data":"0b8ae379007611e7ec918ce7d602bd0090aa469a7d1d5ac396f91047bd4e1ccd"} Nov 03 01:39:05.218082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:05.217908 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b8ae379007611e7ec918ce7d602bd0090aa469a7d1d5ac396f91047bd4e1ccd" Nov 03 01:39:05.218082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:05.217869 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb" Nov 03 01:39:12.159167 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:39:12.159126 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:39:35.538863 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:35.538781 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:39:35.540133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:35.539690 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs" podUID="c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:39:42.156033 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:39:42.155992 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:39:43.258825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.258776 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q"] Nov 03 01:39:43.259353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.259287 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2a51d5af-f8c6-4df0-a909-f2f663c423c9" containerName="revoke-keys" Nov 03 01:39:43.259353 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.259299 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a51d5af-f8c6-4df0-a909-f2f663c423c9" containerName="revoke-keys" Nov 03 01:39:43.259476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.259399 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a51d5af-f8c6-4df0-a909-f2f663c423c9" containerName="revoke-keys" Nov 03 01:39:43.262559 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.262534 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.318068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.318038 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q"] Nov 03 01:39:43.363709 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.363675 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:39:43.363898 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.363749 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:39:43.397457 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.397414 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.397691 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.397573 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcs54\" (UniqueName: \"kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.397778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.397750 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.499129 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.499087 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.499343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.499179 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcs54\" (UniqueName: \"kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.499343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.499239 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.500020 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.499892 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.501709 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.501687 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.556616 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.556536 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcs54\" (UniqueName: \"kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54\") pod \"maas-api-revoke-sub-e2fc9c5b-rlv2q\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.576571 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.576526 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:43.949932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:43.949903 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q"] Nov 03 01:39:43.951080 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:39:43.951053 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f71019_b864_42c9_935a_69223e059880.slice/crio-edca17a7d4f51bc377e01b4ca800cd9a10cbc0add95210ae58ae84e7a73b6705 WatchSource:0}: Error finding container edca17a7d4f51bc377e01b4ca800cd9a10cbc0add95210ae58ae84e7a73b6705: Status 404 returned error can't find the container with id edca17a7d4f51bc377e01b4ca800cd9a10cbc0add95210ae58ae84e7a73b6705 Nov 03 01:39:44.461702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:44.461651 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" event={"ID":"06f71019-b864-42c9-935a-69223e059880","Type":"ContainerStarted","Data":"f1a201e17db81cbbd08e14d9ec7b72a86f60be9e3c735f35e2cb116ef6b465ac"} Nov 03 01:39:44.462189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:44.461727 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" event={"ID":"06f71019-b864-42c9-935a-69223e059880","Type":"ContainerStarted","Data":"edca17a7d4f51bc377e01b4ca800cd9a10cbc0add95210ae58ae84e7a73b6705"} Nov 03 01:39:44.575987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:44.575929 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" podStartSLOduration=1.5759137189999999 podStartE2EDuration="1.575913719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:44.573542624 +0000 UTC m=+5674.719237101" watchObservedRunningTime="2026-11-03 01:39:44.575913719 +0000 UTC m=+5674.721608175" Nov 03 01:39:45.468795 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:45.468758 5144 generic.go:356] "Generic (PLEG): container finished" podID="06f71019-b864-42c9-935a-69223e059880" containerID="f1a201e17db81cbbd08e14d9ec7b72a86f60be9e3c735f35e2cb116ef6b465ac" exitCode=0 Nov 03 01:39:45.469217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:45.468795 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" event={"ID":"06f71019-b864-42c9-935a-69223e059880","Type":"ContainerDied","Data":"f1a201e17db81cbbd08e14d9ec7b72a86f60be9e3c735f35e2cb116ef6b465ac"} Nov 03 01:39:46.616470 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.616444 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:46.731792 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.731705 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca\") pod \"06f71019-b864-42c9-935a-69223e059880\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " Nov 03 01:39:46.731792 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.731786 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca\") pod \"06f71019-b864-42c9-935a-69223e059880\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " Nov 03 01:39:46.732056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.731938 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54\" (UniqueName: \"kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54\") pod \"06f71019-b864-42c9-935a-69223e059880\" (UID: \"06f71019-b864-42c9-935a-69223e059880\") " Nov 03 01:39:46.732164 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.732139 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca" pod "06f71019-b864-42c9-935a-69223e059880" (UID: "06f71019-b864-42c9-935a-69223e059880"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:46.732466 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.732431 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06f71019-b864-42c9-935a-69223e059880-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:46.735017 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.734971 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54" pod "06f71019-b864-42c9-935a-69223e059880" (UID: "06f71019-b864-42c9-935a-69223e059880"). InnerVolumeSpecName "kube-api-access-mcs54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:46.736783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.736754 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca" pod "06f71019-b864-42c9-935a-69223e059880" (UID: "06f71019-b864-42c9-935a-69223e059880"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:46.833256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.833210 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06f71019-b864-42c9-935a-69223e059880-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:46.833256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:46.833249 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcs54\" (UniqueName: \"kubernetes.io/projected/06f71019-b864-42c9-935a-69223e059880-kube-api-access-mcs54\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:47.485410 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:47.485376 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" Nov 03 01:39:47.485607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:47.485407 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q" event={"ID":"06f71019-b864-42c9-935a-69223e059880","Type":"ContainerDied","Data":"edca17a7d4f51bc377e01b4ca800cd9a10cbc0add95210ae58ae84e7a73b6705"} Nov 03 01:39:47.485607 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:47.485456 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edca17a7d4f51bc377e01b4ca800cd9a10cbc0add95210ae58ae84e7a73b6705" Nov 03 01:39:54.044279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.044234 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx"] Nov 03 01:39:54.044887 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.044870 5144 cpu_manager.go:414] "containerMap: removing container" podUID="06f71019-b864-42c9-935a-69223e059880" containerName="revoke-keys" Nov 03 01:39:54.044953 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.044889 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06f71019-b864-42c9-935a-69223e059880" containerName="revoke-keys" Nov 03 01:39:54.045016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.044984 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="06f71019-b864-42c9-935a-69223e059880" containerName="revoke-keys" Nov 03 01:39:54.048568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.048543 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.106130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.106090 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx"] Nov 03 01:39:54.150404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.150335 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:39:54.150601 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.150411 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:39:54.208787 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.208740 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw7zc\" (UniqueName: \"kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.208993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.208892 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.208993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.208932 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.310001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.309893 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw7zc\" (UniqueName: \"kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.310001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.309992 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.310264 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.310029 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.310763 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.310740 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.312544 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.312518 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.366265 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.366223 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw7zc\" (UniqueName: \"kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc\") pod \"maas-api-revoke-sub-5a29efb3-fklsx\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:54.663821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:54.663727 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:55.039294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:55.039266 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx"] Nov 03 01:39:55.040824 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:39:55.040790 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c1a557_b170_41dc_b76c_a204ad705b33.slice/crio-aaf2d48c356e11ca4dba66e4646215c66fc56c90a62d3992baa37bbb19f3ecb6 WatchSource:0}: Error finding container aaf2d48c356e11ca4dba66e4646215c66fc56c90a62d3992baa37bbb19f3ecb6: Status 404 returned error can't find the container with id aaf2d48c356e11ca4dba66e4646215c66fc56c90a62d3992baa37bbb19f3ecb6 Nov 03 01:39:55.533121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:55.533084 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" event={"ID":"a3c1a557-b170-41dc-b76c-a204ad705b33","Type":"ContainerStarted","Data":"218478d41708b109afb729c0e6e8ffa21335a1672040c415a8e26fcdcfc4a3fd"} Nov 03 01:39:55.533540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:55.533138 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" event={"ID":"a3c1a557-b170-41dc-b76c-a204ad705b33","Type":"ContainerStarted","Data":"aaf2d48c356e11ca4dba66e4646215c66fc56c90a62d3992baa37bbb19f3ecb6"} Nov 03 01:39:55.646820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:55.646711 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" podStartSLOduration=1.6466946500000001 podStartE2EDuration="1.64669465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:39:55.645097096 +0000 UTC m=+5685.790791585" watchObservedRunningTime="2026-11-03 01:39:55.64669465 +0000 UTC m=+5685.792389105" Nov 03 01:39:56.541723 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:56.541670 5144 generic.go:356] "Generic (PLEG): container finished" podID="a3c1a557-b170-41dc-b76c-a204ad705b33" containerID="218478d41708b109afb729c0e6e8ffa21335a1672040c415a8e26fcdcfc4a3fd" exitCode=0 Nov 03 01:39:56.542288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:56.541785 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" event={"ID":"a3c1a557-b170-41dc-b76c-a204ad705b33","Type":"ContainerDied","Data":"218478d41708b109afb729c0e6e8ffa21335a1672040c415a8e26fcdcfc4a3fd"} Nov 03 01:39:58.003927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.003892 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:58.093660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.092888 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca\") pod \"a3c1a557-b170-41dc-b76c-a204ad705b33\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " Nov 03 01:39:58.093660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.092955 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc\" (UniqueName: \"kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc\") pod \"a3c1a557-b170-41dc-b76c-a204ad705b33\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " Nov 03 01:39:58.093660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.093040 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca\") pod \"a3c1a557-b170-41dc-b76c-a204ad705b33\" (UID: \"a3c1a557-b170-41dc-b76c-a204ad705b33\") " Nov 03 01:39:58.093660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.093265 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca" pod "a3c1a557-b170-41dc-b76c-a204ad705b33" (UID: "a3c1a557-b170-41dc-b76c-a204ad705b33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:39:58.094217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.094195 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:58.096036 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.096013 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca" pod "a3c1a557-b170-41dc-b76c-a204ad705b33" (UID: "a3c1a557-b170-41dc-b76c-a204ad705b33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:39:58.096036 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.096025 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc" pod "a3c1a557-b170-41dc-b76c-a204ad705b33" (UID: "a3c1a557-b170-41dc-b76c-a204ad705b33"). InnerVolumeSpecName "kube-api-access-tw7zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:39:58.196078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.195980 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw7zc\" (UniqueName: \"kubernetes.io/projected/a3c1a557-b170-41dc-b76c-a204ad705b33-kube-api-access-tw7zc\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:58.196328 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.196307 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c1a557-b170-41dc-b76c-a204ad705b33-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:39:58.571663 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.571598 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" event={"ID":"a3c1a557-b170-41dc-b76c-a204ad705b33","Type":"ContainerDied","Data":"aaf2d48c356e11ca4dba66e4646215c66fc56c90a62d3992baa37bbb19f3ecb6"} Nov 03 01:39:58.571956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.571774 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx" Nov 03 01:39:58.587731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:39:58.571931 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaf2d48c356e11ca4dba66e4646215c66fc56c90a62d3992baa37bbb19f3ecb6" Nov 03 01:40:00.047924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:00.047887 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5"] Nov 03 01:40:00.094139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:00.094105 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2547622-5nlz5"] Nov 03 01:40:01.994826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:01.994785 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c146869-7021-4b25-b319-10fc2f968b82" path="/var/lib/kubelet/pods/7c146869-7021-4b25-b319-10fc2f968b82/volumes" Nov 03 01:40:10.755892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.755853 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb"] Nov 03 01:40:10.787369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.787335 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zptgb"] Nov 03 01:40:10.874506 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.874468 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45"] Nov 03 01:40:10.874946 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.874932 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a3c1a557-b170-41dc-b76c-a204ad705b33" containerName="revoke-keys" Nov 03 01:40:10.875004 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.874948 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c1a557-b170-41dc-b76c-a204ad705b33" containerName="revoke-keys" Nov 03 01:40:10.875054 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.875045 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c1a557-b170-41dc-b76c-a204ad705b33" containerName="revoke-keys" Nov 03 01:40:10.878257 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.878239 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:10.930709 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.930678 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:40:10.931019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.930995 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:40:10.933612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:10.933588 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45"] Nov 03 01:40:11.024682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.024560 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.024682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.024614 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.024682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.024661 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcdp5\" (UniqueName: \"kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.126193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.126160 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.126508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.126219 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.126508 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.126247 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcdp5\" (UniqueName: \"kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.127009 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.126988 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.129121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.129088 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.183239 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.183210 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcdp5\" (UniqueName: \"kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5\") pod \"maas-api-revoke-sub-3d4156f1-ljl45\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.191025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.190997 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:11.320716 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.317905 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:40:11.344978 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.344943 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:40:11.572302 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.572232 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45"] Nov 03 01:40:11.573305 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:40:11.573275 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7bbbfde_77aa_4f46_afa5_11db8e0d6ea3.slice/crio-b33e3eb7d52c0b35cb2a8015c8ba814115b78071fffbf2549ec31cea96fc4f95 WatchSource:0}: Error finding container b33e3eb7d52c0b35cb2a8015c8ba814115b78071fffbf2549ec31cea96fc4f95: Status 404 returned error can't find the container with id b33e3eb7d52c0b35cb2a8015c8ba814115b78071fffbf2549ec31cea96fc4f95 Nov 03 01:40:11.653328 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.653285 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" event={"ID":"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3","Type":"ContainerStarted","Data":"b33e3eb7d52c0b35cb2a8015c8ba814115b78071fffbf2549ec31cea96fc4f95"} Nov 03 01:40:11.994982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:11.994938 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a51d5af-f8c6-4df0-a909-f2f663c423c9" path="/var/lib/kubelet/pods/2a51d5af-f8c6-4df0-a909-f2f663c423c9/volumes" Nov 03 01:40:12.157688 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:40:12.157573 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:40:12.661063 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:12.661030 5144 generic.go:356] "Generic (PLEG): container finished" podID="c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" containerID="5bba89ab32ed1a1dc7d587d11fc862214788a3c475072c90abb86f919075c424" exitCode=0 Nov 03 01:40:12.661253 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:12.661080 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" event={"ID":"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3","Type":"ContainerDied","Data":"5bba89ab32ed1a1dc7d587d11fc862214788a3c475072c90abb86f919075c424"} Nov 03 01:40:13.805816 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.805791 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:13.860209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.860180 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca\") pod \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " Nov 03 01:40:13.860394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.860246 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca\") pod \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " Nov 03 01:40:13.860394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.860365 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5\" (UniqueName: \"kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5\") pod \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\" (UID: \"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3\") " Nov 03 01:40:13.860652 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.860612 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca" pod "c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" (UID: "c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:40:13.860843 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.860823 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:40:13.862606 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.862573 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca" pod "c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" (UID: "c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:40:13.862743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.862664 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5" pod "c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" (UID: "c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3"). InnerVolumeSpecName "kube-api-access-qcdp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:40:13.961477 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.961402 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:40:13.961477 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:13.961428 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcdp5\" (UniqueName: \"kubernetes.io/projected/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3-kube-api-access-qcdp5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:40:14.674229 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:14.674190 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" Nov 03 01:40:14.674435 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:14.674200 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45" event={"ID":"c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3","Type":"ContainerDied","Data":"b33e3eb7d52c0b35cb2a8015c8ba814115b78071fffbf2549ec31cea96fc4f95"} Nov 03 01:40:14.674435 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:14.674326 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b33e3eb7d52c0b35cb2a8015c8ba814115b78071fffbf2549ec31cea96fc4f95" Nov 03 01:40:27.357720 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.357035 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr"] Nov 03 01:40:27.358381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.357934 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" containerName="revoke-keys" Nov 03 01:40:27.358381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.357956 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" containerName="revoke-keys" Nov 03 01:40:27.358381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.358083 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" containerName="revoke-keys" Nov 03 01:40:27.362661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.362244 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.481656 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.481077 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr"] Nov 03 01:40:27.507760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.506716 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b19258-openshift-default-dockercfg-dvmd7\"" Nov 03 01:40:27.507760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507523 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8dh\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.507760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507583 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.507760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507616 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.507760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507663 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.507760 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507708 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.508279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507880 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.508279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.507939 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.508279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.508020 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.508279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.508062 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610401 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8dh\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610478 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610511 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610536 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610576 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610620 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610684 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610766 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.610802 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.612005 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.613256 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.613471 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.613714 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.630710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.613915 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.632246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.632181 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.654350 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.654251 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.670500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.670423 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.703517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.695804 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8dh\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh\") pod \"e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:27.703517 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:27.699811 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:28.148816 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:28.148785 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr"] Nov 03 01:40:28.149735 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:40:28.149711 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda22f94a8_4c31_4ae6_9a45_ff6fb1748ca2.slice/crio-447276f2e039e595c024b59d9a3f4dc74bec574761da84832f8872c73b3600fe WatchSource:0}: Error finding container 447276f2e039e595c024b59d9a3f4dc74bec574761da84832f8872c73b3600fe: Status 404 returned error can't find the container with id 447276f2e039e595c024b59d9a3f4dc74bec574761da84832f8872c73b3600fe Nov 03 01:40:28.152145 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:28.152108 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:40:28.152240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:28.152196 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:40:28.152286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:28.152240 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:40:28.814748 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:28.814706 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" event={"ID":"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2","Type":"ContainerStarted","Data":"6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b"} Nov 03 01:40:28.815192 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:28.814801 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" event={"ID":"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2","Type":"ContainerStarted","Data":"447276f2e039e595c024b59d9a3f4dc74bec574761da84832f8872c73b3600fe"} Nov 03 01:40:29.030267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:29.030211 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podStartSLOduration=2.030196464 podStartE2EDuration="2.030196464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:28.978875685 +0000 UTC m=+5719.124570138" watchObservedRunningTime="2026-11-03 01:40:29.030196464 +0000 UTC m=+5719.175890968" Nov 03 01:40:29.704528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:29.704482 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:29.710217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:29.710184 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:29.821107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:29.821076 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:29.822120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:29.822099 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:31.051167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:31.051136 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk"] Nov 03 01:40:31.100813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:31.100777 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-qm8mk"] Nov 03 01:40:31.995532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:31.995496 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c247be5d-763f-4594-8b64-f5ecf51a3049" path="/var/lib/kubelet/pods/c247be5d-763f-4594-8b64-f5ecf51a3049/volumes" Nov 03 01:40:35.551646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.551597 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j"] Nov 03 01:40:35.556538 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.556515 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.582165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.582130 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnbb8\" (UniqueName: \"kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8\") pod \"payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.582382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.582324 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.610984 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.610948 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j"] Nov 03 01:40:35.656822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.656788 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b19258\"" Nov 03 01:40:35.656822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.656808 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-ad5e139a-dockercfg-dznff\"" Nov 03 01:40:35.683344 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.683302 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.683551 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.683399 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnbb8\" (UniqueName: \"kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8\") pod \"payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.684153 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.684131 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.709697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.709663 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b"] Nov 03 01:40:35.720056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.720027 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:35.739834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.739800 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnbb8\" (UniqueName: \"kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8\") pod \"payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.765158 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.765125 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b"] Nov 03 01:40:35.784742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.784686 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:35.784742 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.784723 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbwz6\" (UniqueName: \"kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6\") pod \"payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:35.871313 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.871222 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:35.886136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.886096 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:35.886136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.886133 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbwz6\" (UniqueName: \"kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6\") pod \"payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:35.886851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.886830 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:35.946475 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:35.946272 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbwz6\" (UniqueName: \"kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6\") pod \"payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:36.033492 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.033457 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:36.246327 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.246301 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j"] Nov 03 01:40:36.247512 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:40:36.247482 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod089c3d0d_3f62_4c3d_add1_40317a3692ab.slice/crio-b8755af0918d448bb4df7bbbe4f6242c9199fb38fa84551c2310f520b9bfaa3e WatchSource:0}: Error finding container b8755af0918d448bb4df7bbbe4f6242c9199fb38fa84551c2310f520b9bfaa3e: Status 404 returned error can't find the container with id b8755af0918d448bb4df7bbbe4f6242c9199fb38fa84551c2310f520b9bfaa3e Nov 03 01:40:36.407741 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.407691 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b"] Nov 03 01:40:36.408687 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:40:36.408625 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40f13fb_ebf3_40c9_85d3_ebc47d88ffa4.slice/crio-bf2b3395515d3de0a1438dea9187996365d161b43947ad1d4142362c278a5698 WatchSource:0}: Error finding container bf2b3395515d3de0a1438dea9187996365d161b43947ad1d4142362c278a5698: Status 404 returned error can't find the container with id bf2b3395515d3de0a1438dea9187996365d161b43947ad1d4142362c278a5698 Nov 03 01:40:36.865733 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.865690 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" event={"ID":"089c3d0d-3f62-4c3d-add1-40317a3692ab","Type":"ContainerStarted","Data":"5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0"} Nov 03 01:40:36.866245 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.865753 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" event={"ID":"089c3d0d-3f62-4c3d-add1-40317a3692ab","Type":"ContainerStarted","Data":"b8755af0918d448bb4df7bbbe4f6242c9199fb38fa84551c2310f520b9bfaa3e"} Nov 03 01:40:36.866245 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.865780 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:36.867424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.867398 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" event={"ID":"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4","Type":"ContainerStarted","Data":"dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f"} Nov 03 01:40:36.867546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.867449 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:36.867546 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.867469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" event={"ID":"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4","Type":"ContainerStarted","Data":"bf2b3395515d3de0a1438dea9187996365d161b43947ad1d4142362c278a5698"} Nov 03 01:40:36.978068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:36.978006 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" podStartSLOduration=1.977990423 podStartE2EDuration="1.977990423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:36.977815945 +0000 UTC m=+5727.123510402" watchObservedRunningTime="2026-11-03 01:40:36.977990423 +0000 UTC m=+5727.123684883" Nov 03 01:40:37.091653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:37.091573 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" podStartSLOduration=2.091557149 podStartE2EDuration="2.091557149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:40:37.089615686 +0000 UTC m=+5727.235310141" watchObservedRunningTime="2026-11-03 01:40:37.091557149 +0000 UTC m=+5727.237251604" Nov 03 01:40:42.155211 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:40:42.155171 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:40:47.877935 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:47.877897 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:40:47.878359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:47.878058 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:40:55.011133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:55.011100 5144 generic.go:356] "Generic (PLEG): container finished" podID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerID="6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b" exitCode=0 Nov 03 01:40:55.011609 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:55.011176 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" event={"ID":"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2","Type":"ContainerDied","Data":"6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b"} Nov 03 01:40:55.012029 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:55.011617 5144 scope.go:122] "RemoveContainer" containerID="6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b" Nov 03 01:40:55.013068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:55.013036 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:40:55.013205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:55.013127 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:40:55.013205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:55.013177 5144 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 01:40:56.041267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:56.041230 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" event={"ID":"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2","Type":"ContainerStarted","Data":"fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398"} Nov 03 01:40:56.045886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:56.045858 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq"] Nov 03 01:40:56.095723 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:56.095681 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-n96rq"] Nov 03 01:40:56.703948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:56.703911 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:40:56.705367 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:56.705340 5144 patch_prober.go:29] interesting pod/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 01:40:56.705542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:56.705408 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 01:40:57.704474 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:57.704433 5144 patch_prober.go:29] interesting pod/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 01:40:57.704934 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:57.704492 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 01:40:57.997123 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:57.997028 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70ac437-31a0-41ac-be02-af0aa2659d62" path="/var/lib/kubelet/pods/c70ac437-31a0-41ac-be02-af0aa2659d62/volumes" Nov 03 01:40:58.704550 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:58.704513 5144 patch_prober.go:29] interesting pod/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 01:40:58.705095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:58.704566 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 01:40:59.704259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:59.704215 5144 patch_prober.go:29] interesting pod/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 01:40:59.704480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:59.704279 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 01:40:59.836661 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:40:59.833740 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:41:00.704557 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:00.704522 5144 patch_prober.go:29] interesting pod/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" start-of-body= Nov 03 01:41:00.704799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:00.704587 5144 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.141:15021/healthz/ready\": dial tcp 10.132.1.141:15021: connect: connection refused" Nov 03 01:41:01.708973 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:01.708929 5144 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:41:01.710106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:01.710082 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:41:09.044583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:09.044545 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw"] Nov 03 01:41:09.094030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:09.093997 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-tr6jw"] Nov 03 01:41:09.995913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:09.995874 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0de3af84-8739-49b4-93ef-fbc7b9133af2" path="/var/lib/kubelet/pods/0de3af84-8739-49b4-93ef-fbc7b9133af2/volumes" Nov 03 01:41:12.155737 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:41:12.155692 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:41:14.884877 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:14.884844 5144 scope.go:122] "RemoveContainer" containerID="38e1ec298023838647eeeb1387e083bc87b58a9a90168e6757a11fdf4f4dd64d" Nov 03 01:41:41.048707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:41.048663 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7"] Nov 03 01:41:41.097621 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:41.097584 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-pc4b7"] Nov 03 01:41:41.996518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:41.996479 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fdfceec-9aa6-4257-9f7d-f7b90a827641" path="/var/lib/kubelet/pods/6fdfceec-9aa6-4257-9f7d-f7b90a827641/volumes" Nov 03 01:41:42.155806 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:41:42.155768 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:41:47.145876 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.145838 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42"] Nov 03 01:41:47.149352 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.149330 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.183284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.183240 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbgsf\" (UniqueName: \"kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.183491 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.183364 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.183564 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.183533 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.202929 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.202895 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:41:47.203181 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.202958 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:41:47.255968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.255922 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42"] Nov 03 01:41:47.284699 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.284663 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbgsf\" (UniqueName: \"kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.284903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.284729 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.284903 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.284854 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.285370 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.285346 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.287404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.287381 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.342168 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.342122 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbgsf\" (UniqueName: \"kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf\") pod \"maas-api-revoke-sub-adbe0b63-hqw42\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.462174 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.462086 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:47.835376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:47.835344 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42"] Nov 03 01:41:47.837090 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:41:47.837047 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78800654_60c1_4906_b3e8_206f615c11c6.slice/crio-bfb93e5e4a354188f6729fe1615f1998e601fc37f3b11a9e232cd223fa266469 WatchSource:0}: Error finding container bfb93e5e4a354188f6729fe1615f1998e601fc37f3b11a9e232cd223fa266469: Status 404 returned error can't find the container with id bfb93e5e4a354188f6729fe1615f1998e601fc37f3b11a9e232cd223fa266469 Nov 03 01:41:48.388753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:48.388620 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" event={"ID":"78800654-60c1-4906-b3e8-206f615c11c6","Type":"ContainerStarted","Data":"b176ee92ac2aab47a4c99d12340c71ed7b2350c7a6d4369b22880847e1c35486"} Nov 03 01:41:48.388753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:48.388716 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" event={"ID":"78800654-60c1-4906-b3e8-206f615c11c6","Type":"ContainerStarted","Data":"bfb93e5e4a354188f6729fe1615f1998e601fc37f3b11a9e232cd223fa266469"} Nov 03 01:41:48.502729 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:48.502674 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" podStartSLOduration=1.50265884 podStartE2EDuration="1.50265884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:41:48.500089521 +0000 UTC m=+5798.645783977" watchObservedRunningTime="2026-11-03 01:41:48.50265884 +0000 UTC m=+5798.648353296" Nov 03 01:41:49.396220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:49.396182 5144 generic.go:356] "Generic (PLEG): container finished" podID="78800654-60c1-4906-b3e8-206f615c11c6" containerID="b176ee92ac2aab47a4c99d12340c71ed7b2350c7a6d4369b22880847e1c35486" exitCode=0 Nov 03 01:41:49.396687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:49.396265 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" event={"ID":"78800654-60c1-4906-b3e8-206f615c11c6","Type":"ContainerDied","Data":"b176ee92ac2aab47a4c99d12340c71ed7b2350c7a6d4369b22880847e1c35486"} Nov 03 01:41:50.557854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.557826 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:50.621450 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.621414 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca\") pod \"78800654-60c1-4906-b3e8-206f615c11c6\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " Nov 03 01:41:50.621675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.621491 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca\") pod \"78800654-60c1-4906-b3e8-206f615c11c6\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " Nov 03 01:41:50.621757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.621705 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf\" (UniqueName: \"kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf\") pod \"78800654-60c1-4906-b3e8-206f615c11c6\" (UID: \"78800654-60c1-4906-b3e8-206f615c11c6\") " Nov 03 01:41:50.621930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.621904 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca" pod "78800654-60c1-4906-b3e8-206f615c11c6" (UID: "78800654-60c1-4906-b3e8-206f615c11c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:41:50.622063 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.622039 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:41:50.623945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.623921 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf" pod "78800654-60c1-4906-b3e8-206f615c11c6" (UID: "78800654-60c1-4906-b3e8-206f615c11c6"). InnerVolumeSpecName "kube-api-access-qbgsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:41:50.624058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.624016 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca" pod "78800654-60c1-4906-b3e8-206f615c11c6" (UID: "78800654-60c1-4906-b3e8-206f615c11c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:41:50.722874 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.722787 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbgsf\" (UniqueName: \"kubernetes.io/projected/78800654-60c1-4906-b3e8-206f615c11c6-kube-api-access-qbgsf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:41:50.722874 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:50.722822 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78800654-60c1-4906-b3e8-206f615c11c6-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:41:51.045287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:51.045248 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf"] Nov 03 01:41:51.049056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:51.049025 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-r9flf"] Nov 03 01:41:51.409512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:51.409419 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" Nov 03 01:41:51.409512 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:51.409416 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42" event={"ID":"78800654-60c1-4906-b3e8-206f615c11c6","Type":"ContainerDied","Data":"bfb93e5e4a354188f6729fe1615f1998e601fc37f3b11a9e232cd223fa266469"} Nov 03 01:41:51.409754 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:51.409540 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfb93e5e4a354188f6729fe1615f1998e601fc37f3b11a9e232cd223fa266469" Nov 03 01:41:51.995857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:41:51.995825 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8278fe4-cfbb-405a-b32f-fe9afbd83eda" path="/var/lib/kubelet/pods/f8278fe4-cfbb-405a-b32f-fe9afbd83eda/volumes" Nov 03 01:42:05.968702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:05.968663 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b"] Nov 03 01:42:06.027853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:06.027815 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j"] Nov 03 01:42:10.969813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:10.969760 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" podUID="b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" containerName="payload-processing" containerID="cri-o://dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f" gracePeriod=25 Nov 03 01:42:11.028613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.028567 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerName="payload-pre-processing" containerID="cri-o://5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0" gracePeriod=25 Nov 03 01:42:11.246243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.246216 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:42:11.355286 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.355250 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume\") pod \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " Nov 03 01:42:11.355569 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.355305 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6\" (UniqueName: \"kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6\") pod \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\" (UID: \"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4\") " Nov 03 01:42:11.355668 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.355647 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume" pod "b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" (UID: "b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:11.357673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.357611 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6" pod "b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" (UID: "b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4"). InnerVolumeSpecName "kube-api-access-qbwz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:11.456391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.456351 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:11.456391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.456381 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbwz6\" (UniqueName: \"kubernetes.io/projected/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4-kube-api-access-qbwz6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:11.540078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.540039 5144 generic.go:356] "Generic (PLEG): container finished" podID="b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" containerID="dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f" exitCode=0 Nov 03 01:42:11.540306 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.540089 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" Nov 03 01:42:11.540306 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.540094 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" event={"ID":"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4","Type":"ContainerDied","Data":"dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f"} Nov 03 01:42:11.540306 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.540157 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b" event={"ID":"b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4","Type":"ContainerDied","Data":"bf2b3395515d3de0a1438dea9187996365d161b43947ad1d4142362c278a5698"} Nov 03 01:42:11.540306 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.540187 5144 scope.go:122] "RemoveContainer" containerID="dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f" Nov 03 01:42:11.561330 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.561303 5144 scope.go:122] "RemoveContainer" containerID="dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f" Nov 03 01:42:11.561730 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:42:11.561707 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f\": container with ID starting with dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f not found: ID does not exist" containerID="dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f" Nov 03 01:42:11.561812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.561739 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f"} err="failed to get container status \"dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f\": rpc error: code = NotFound desc = could not find container \"dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f\": container with ID starting with dff4c52a481a154ff2f2d53d16811e17a73c1c671b93fe55cf21ce88ab5ae43f not found: ID does not exist" Nov 03 01:42:11.704137 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.704047 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b"] Nov 03 01:42:11.707269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.707235 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b19258-679c5d8fd5985b"] Nov 03 01:42:11.998019 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:11.997915 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" path="/var/lib/kubelet/pods/b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4/volumes" Nov 03 01:42:12.156563 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:42:12.156522 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:42:14.910397 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:14.910359 5144 scope.go:122] "RemoveContainer" containerID="c4923f2d79750f1716c30b0ecf1e69d63d2b6042d7f3279497aaf6677806dcaf" Nov 03 01:42:14.919979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:14.919958 5144 scope.go:122] "RemoveContainer" containerID="79f8ca5b2c56b529e755e5562501065591fc087142bdcd1cf833c88e9bb73448" Nov 03 01:42:14.929277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:14.929253 5144 scope.go:122] "RemoveContainer" containerID="a0a55e71e56d643788c30f951690adeb5c7ac7526774211708a3161c97da9223" Nov 03 01:42:17.875830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:17.875786 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:25.044674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:25.044626 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st"] Nov 03 01:42:25.094408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:25.094371 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-n29st"] Nov 03 01:42:25.995883 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:25.995839 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a59b80a-3ff2-4348-b248-0304f9cd5189" path="/var/lib/kubelet/pods/4a59b80a-3ff2-4348-b248-0304f9cd5189/volumes" Nov 03 01:42:27.876311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:27.876268 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 01:42:31.289725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.289648 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:42:31.356674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.356624 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume\") pod \"089c3d0d-3f62-4c3d-add1-40317a3692ab\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " Nov 03 01:42:31.356878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.356699 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8\" (UniqueName: \"kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8\") pod \"089c3d0d-3f62-4c3d-add1-40317a3692ab\" (UID: \"089c3d0d-3f62-4c3d-add1-40317a3692ab\") " Nov 03 01:42:31.357117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.357082 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume" pod "089c3d0d-3f62-4c3d-add1-40317a3692ab" (UID: "089c3d0d-3f62-4c3d-add1-40317a3692ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:31.359072 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.359049 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8" pod "089c3d0d-3f62-4c3d-add1-40317a3692ab" (UID: "089c3d0d-3f62-4c3d-add1-40317a3692ab"). InnerVolumeSpecName "kube-api-access-wnbb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:31.458180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.458144 5144 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/089c3d0d-3f62-4c3d-add1-40317a3692ab-plugins-config-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:31.458180 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.458179 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnbb8\" (UniqueName: \"kubernetes.io/projected/089c3d0d-3f62-4c3d-add1-40317a3692ab-kube-api-access-wnbb8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:31.672278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.672180 5144 generic.go:356] "Generic (PLEG): container finished" podID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerID="5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0" exitCode=0 Nov 03 01:42:31.672278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.672231 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" Nov 03 01:42:31.672525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.672269 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" event={"ID":"089c3d0d-3f62-4c3d-add1-40317a3692ab","Type":"ContainerDied","Data":"5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0"} Nov 03 01:42:31.672525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.672328 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j" event={"ID":"089c3d0d-3f62-4c3d-add1-40317a3692ab","Type":"ContainerDied","Data":"b8755af0918d448bb4df7bbbe4f6242c9199fb38fa84551c2310f520b9bfaa3e"} Nov 03 01:42:31.672525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.672356 5144 scope.go:122] "RemoveContainer" containerID="5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0" Nov 03 01:42:31.683345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.683324 5144 scope.go:122] "RemoveContainer" containerID="5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0" Nov 03 01:42:31.683721 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:42:31.683696 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0\": container with ID starting with 5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0 not found: ID does not exist" containerID="5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0" Nov 03 01:42:31.683784 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.683734 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0"} err="failed to get container status \"5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0\": rpc error: code = NotFound desc = could not find container \"5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0\": container with ID starting with 5388d12bb3b62ef10198d60be50f0b1b06d35e9d10c914400888a3d270081cf0 not found: ID does not exist" Nov 03 01:42:31.837697 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.837663 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j"] Nov 03 01:42:31.887416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.887376 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b19258-756c49d42j"] Nov 03 01:42:31.995812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:31.995727 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" path="/var/lib/kubelet/pods/089c3d0d-3f62-4c3d-add1-40317a3692ab/volumes" Nov 03 01:42:42.155098 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:42:42.155058 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:42:43.002455 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:43.002416 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr"] Nov 03 01:42:43.002756 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:43.002673 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" containerID="cri-o://fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398" gracePeriod=30 Nov 03 01:42:44.559776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.559736 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45"] Nov 03 01:42:44.609580 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.609541 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-ljl45"] Nov 03 01:42:44.685520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.685486 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d"] Nov 03 01:42:44.685983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.685969 5144 cpu_manager.go:414] "containerMap: removing container" podUID="b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" containerName="payload-processing" Nov 03 01:42:44.686056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.685984 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" containerName="payload-processing" Nov 03 01:42:44.686056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.685998 5144 cpu_manager.go:414] "containerMap: removing container" podUID="78800654-60c1-4906-b3e8-206f615c11c6" containerName="revoke-keys" Nov 03 01:42:44.686056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.686002 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78800654-60c1-4906-b3e8-206f615c11c6" containerName="revoke-keys" Nov 03 01:42:44.686056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.686014 5144 cpu_manager.go:414] "containerMap: removing container" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerName="payload-pre-processing" Nov 03 01:42:44.686056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.686018 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerName="payload-pre-processing" Nov 03 01:42:44.686317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.686107 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40f13fb-ebf3-40c9-85d3-ebc47d88ffa4" containerName="payload-processing" Nov 03 01:42:44.686317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.686117 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="78800654-60c1-4906-b3e8-206f615c11c6" containerName="revoke-keys" Nov 03 01:42:44.686317 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.686122 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="089c3d0d-3f62-4c3d-add1-40317a3692ab" containerName="payload-pre-processing" Nov 03 01:42:44.690602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.690582 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.743038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.743007 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:42:44.743278 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.743255 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:42:44.745854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.745821 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d"] Nov 03 01:42:44.802048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.801997 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.802798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.802770 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.802929 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.802818 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.904134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.904033 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.904134 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.904101 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.904368 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.904182 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.904913 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.904887 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.906860 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.906837 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:44.963736 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:44.963696 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf\") pod \"maas-api-revoke-sub-3d4156f1-b4n9d\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:45.004974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:45.004938 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:45.378321 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:45.378290 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d"] Nov 03 01:42:45.380174 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:42:45.380146 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce5ac2f_9c1e_48da_b458_54e76b8ce391.slice/crio-3cebb426c50a7422684c365134682a4243ec0f4398f154d05cee8b88c51a6c94 WatchSource:0}: Error finding container 3cebb426c50a7422684c365134682a4243ec0f4398f154d05cee8b88c51a6c94: Status 404 returned error can't find the container with id 3cebb426c50a7422684c365134682a4243ec0f4398f154d05cee8b88c51a6c94 Nov 03 01:42:45.769077 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:45.769041 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" event={"ID":"5ce5ac2f-9c1e-48da-b458-54e76b8ce391","Type":"ContainerStarted","Data":"50a8ab4fd034e868a48c4c05223c1f58986d34fe7db1bdf8ccb26364e78713ea"} Nov 03 01:42:45.769489 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:45.769098 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" event={"ID":"5ce5ac2f-9c1e-48da-b458-54e76b8ce391","Type":"ContainerStarted","Data":"3cebb426c50a7422684c365134682a4243ec0f4398f154d05cee8b88c51a6c94"} Nov 03 01:42:45.996091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:45.996055 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3" path="/var/lib/kubelet/pods/c7bbbfde-77aa-4f46-afa5-11db8e0d6ea3/volumes" Nov 03 01:42:46.712584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:46.712545 5144 patch_prober.go:29] interesting pod/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 01:42:46.712812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:46.712658 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 01:42:46.777107 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:46.777072 5144 generic.go:356] "Generic (PLEG): container finished" podID="5ce5ac2f-9c1e-48da-b458-54e76b8ce391" containerID="50a8ab4fd034e868a48c4c05223c1f58986d34fe7db1bdf8ccb26364e78713ea" exitCode=0 Nov 03 01:42:46.777537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:46.777143 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" event={"ID":"5ce5ac2f-9c1e-48da-b458-54e76b8ce391","Type":"ContainerDied","Data":"50a8ab4fd034e868a48c4c05223c1f58986d34fe7db1bdf8ccb26364e78713ea"} Nov 03 01:42:47.928958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:47.928928 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:48.037583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.037556 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf\") pod \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " Nov 03 01:42:48.037730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.037673 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca\") pod \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " Nov 03 01:42:48.037730 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.037713 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca\") pod \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\" (UID: \"5ce5ac2f-9c1e-48da-b458-54e76b8ce391\") " Nov 03 01:42:48.038142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.038113 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca" pod "5ce5ac2f-9c1e-48da-b458-54e76b8ce391" (UID: "5ce5ac2f-9c1e-48da-b458-54e76b8ce391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:48.039854 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.039830 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca" pod "5ce5ac2f-9c1e-48da-b458-54e76b8ce391" (UID: "5ce5ac2f-9c1e-48da-b458-54e76b8ce391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:42:48.039966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.039945 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf" pod "5ce5ac2f-9c1e-48da-b458-54e76b8ce391" (UID: "5ce5ac2f-9c1e-48da-b458-54e76b8ce391"). InnerVolumeSpecName "kube-api-access-wxldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:48.139534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.139500 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.139534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.139529 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxldf\" (UniqueName: \"kubernetes.io/projected/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-kube-api-access-wxldf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.139534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.139539 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce5ac2f-9c1e-48da-b458-54e76b8ce391-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.271445 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.271413 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:42:48.442066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.441970 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442066 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442011 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442083 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442125 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442150 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442183 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442216 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442239 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442297 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy\") pod \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\" (UID: \"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2\") " Nov 03 01:42:48.442622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442303 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:48.442622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442397 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:48.442622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442543 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:48.442822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442701 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:48.442822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442797 5144 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-data\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.442994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442823 5144 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-certs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.442994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442837 5144 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-credential-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.442994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442850 5144 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-workload-socket\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.442994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.442793 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:42:48.444622 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.444597 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:42:48.444759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.444721 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 01:42:48.444759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.444731 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:48.444759 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.444739 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh" pod "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" (UID: "a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2"). InnerVolumeSpecName "kube-api-access-fq8dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:42:48.543664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.543591 5144 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-token\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.543664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.543662 5144 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-podinfo\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.543664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.543672 5144 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istiod-ca-cert\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.543970 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.543683 5144 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-istio-envoy\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.543970 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.543693 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8dh\" (UniqueName: \"kubernetes.io/projected/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2-kube-api-access-fq8dh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:42:48.790513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.790476 5144 generic.go:356] "Generic (PLEG): container finished" podID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerID="fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398" exitCode=0 Nov 03 01:42:48.790744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.790521 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" Nov 03 01:42:48.790744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.790552 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" event={"ID":"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2","Type":"ContainerDied","Data":"fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398"} Nov 03 01:42:48.790744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.790621 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr" event={"ID":"a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2","Type":"ContainerDied","Data":"447276f2e039e595c024b59d9a3f4dc74bec574761da84832f8872c73b3600fe"} Nov 03 01:42:48.790744 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.790687 5144 scope.go:122] "RemoveContainer" containerID="fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398" Nov 03 01:42:48.792249 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.792225 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" event={"ID":"5ce5ac2f-9c1e-48da-b458-54e76b8ce391","Type":"ContainerDied","Data":"3cebb426c50a7422684c365134682a4243ec0f4398f154d05cee8b88c51a6c94"} Nov 03 01:42:48.792367 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.792267 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cebb426c50a7422684c365134682a4243ec0f4398f154d05cee8b88c51a6c94" Nov 03 01:42:48.792367 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.792300 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d" Nov 03 01:42:48.803344 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.803324 5144 scope.go:122] "RemoveContainer" containerID="6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b" Nov 03 01:42:48.813117 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.813096 5144 scope.go:122] "RemoveContainer" containerID="fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398" Nov 03 01:42:48.813429 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:42:48.813407 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398\": container with ID starting with fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398 not found: ID does not exist" containerID="fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398" Nov 03 01:42:48.813513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.813438 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398"} err="failed to get container status \"fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398\": rpc error: code = NotFound desc = could not find container \"fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398\": container with ID starting with fa2d0edaea167287863c571e8751a22a0bac1f8fbcd616427d7a87dc67a41398 not found: ID does not exist" Nov 03 01:42:48.813513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.813459 5144 scope.go:122] "RemoveContainer" containerID="6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b" Nov 03 01:42:48.813776 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:42:48.813736 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b\": container with ID starting with 6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b not found: ID does not exist" containerID="6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b" Nov 03 01:42:48.813776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:48.813765 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b"} err="failed to get container status \"6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b\": rpc error: code = NotFound desc = could not find container \"6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b\": container with ID starting with 6606f3c065b90d0de87a748161acb6adb3ad4da4aeba9373cad4ba1230c0a45b not found: ID does not exist" Nov 03 01:42:49.006161 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:49.006122 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr"] Nov 03 01:42:49.009774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:49.009746 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b19258-openshift-default-cb589d597-zcjzr"] Nov 03 01:42:49.994965 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:49.994931 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" path="/var/lib/kubelet/pods/a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2/volumes" Nov 03 01:42:51.047195 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:51.047157 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd"] Nov 03 01:42:51.097183 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:51.097143 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-cvcfd"] Nov 03 01:42:51.995688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:42:51.995624 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8efb5d90-e3a5-4597-be78-ef818d3ade4c" path="/var/lib/kubelet/pods/8efb5d90-e3a5-4597-be78-ef818d3ade4c/volumes" Nov 03 01:43:03.449674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:03.449613 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5"] Nov 03 01:43:03.450171 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:03.449867 5144 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" podUID="4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" containerName="manager" containerID="cri-o://5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5" gracePeriod=10 Nov 03 01:43:04.809386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.809356 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 01:43:04.823576 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.823547 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7\" (UniqueName: \"kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7\") pod \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " Nov 03 01:43:04.823770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.823715 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume\") pod \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\" (UID: \"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c\") " Nov 03 01:43:04.824092 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.824068 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume" pod "4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" (UID: "4eb2b0f0-fb57-4948-b3a2-4d9094b4992c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 01:43:04.825883 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.825854 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7" pod "4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" (UID: "4eb2b0f0-fb57-4948-b3a2-4d9094b4992c"). InnerVolumeSpecName "kube-api-access-zjpb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:43:04.923835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.923743 5144 generic.go:356] "Generic (PLEG): container finished" podID="4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" containerID="5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5" exitCode=0 Nov 03 01:43:04.923835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.923782 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" Nov 03 01:43:04.923835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.923805 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" event={"ID":"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c","Type":"ContainerDied","Data":"5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5"} Nov 03 01:43:04.924138 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.923902 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5" event={"ID":"4eb2b0f0-fb57-4948-b3a2-4d9094b4992c","Type":"ContainerDied","Data":"652937fdddaf35d8f8d7d4a50709a3b20a14bdd0c4d3aa1ff1b351a7c44da2c4"} Nov 03 01:43:04.924138 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.923932 5144 scope.go:122] "RemoveContainer" containerID="5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5" Nov 03 01:43:04.924496 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.924480 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjpb7\" (UniqueName: \"kubernetes.io/projected/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-kube-api-access-zjpb7\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:43:04.924555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.924500 5144 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c-extensions-socket-volume\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:43:04.943086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.943064 5144 scope.go:122] "RemoveContainer" containerID="5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5" Nov 03 01:43:04.943380 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:43:04.943355 5144 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5\": container with ID starting with 5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5 not found: ID does not exist" containerID="5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5" Nov 03 01:43:04.943472 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:04.943387 5144 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5"} err="failed to get container status \"5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5\": rpc error: code = NotFound desc = could not find container \"5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5\": container with ID starting with 5cf6daa974967eb24e48b4bb0ca27175283c4e5491273a4a689dbeb6c8d96be5 not found: ID does not exist" Nov 03 01:43:05.085224 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:05.085187 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5"] Nov 03 01:43:05.089050 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:05.089018 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-wkpw5"] Nov 03 01:43:05.995425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:05.995393 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" path="/var/lib/kubelet/pods/4eb2b0f0-fb57-4948-b3a2-4d9094b4992c/volumes" Nov 03 01:43:12.155088 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:43:12.155044 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:43:13.043509 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:13.043472 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt"] Nov 03 01:43:13.093260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:13.093228 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-zfjgt"] Nov 03 01:43:13.995214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:13.995178 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c1934e-91bb-45dc-9683-1ed5de03c6a5" path="/var/lib/kubelet/pods/73c1934e-91bb-45dc-9683-1ed5de03c6a5/volumes" Nov 03 01:43:14.966537 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:14.966506 5144 scope.go:122] "RemoveContainer" containerID="66b87ab02ad1ce2437e3553f8a599d1f0ec3f5b21e0433bd2454f606e57ebd43" Nov 03 01:43:14.976103 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:14.976078 5144 scope.go:122] "RemoveContainer" containerID="648822f980af55f448c9b4b33bb42dbc4500b69194e3ce518a939934db508b33" Nov 03 01:43:30.044604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:30.044512 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5"] Nov 03 01:43:30.094092 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:30.094053 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-vdkb5"] Nov 03 01:43:31.996005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:43:31.995960 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18" path="/var/lib/kubelet/pods/8b9caccb-6d8f-4b51-9cf3-759e9e0f0b18/volumes" Nov 03 01:43:42.156200 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:43:42.156150 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:44:06.991812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.991777 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg"] Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992211 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992222 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992237 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992242 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992248 5144 cpu_manager.go:414] "containerMap: removing container" podUID="5ce5ac2f-9c1e-48da-b458-54e76b8ce391" containerName="revoke-keys" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992253 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce5ac2f-9c1e-48da-b458-54e76b8ce391" containerName="revoke-keys" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992267 5144 cpu_manager.go:414] "containerMap: removing container" podUID="4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" containerName="manager" Nov 03 01:44:06.992311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992272 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" containerName="manager" Nov 03 01:44:06.992575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992341 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" Nov 03 01:44:06.992575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992351 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a22f94a8-4c31-4ae6-9a45-ff6fb1748ca2" containerName="istio-proxy" Nov 03 01:44:06.992575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992356 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce5ac2f-9c1e-48da-b458-54e76b8ce391" containerName="revoke-keys" Nov 03 01:44:06.992575 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.992366 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eb2b0f0-fb57-4948-b3a2-4d9094b4992c" containerName="manager" Nov 03 01:44:06.995616 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:06.995599 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.049106 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.049074 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg"] Nov 03 01:44:07.095783 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.095746 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-5fwfj\"" Nov 03 01:44:07.160335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.160301 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qhl2\" (UniqueName: \"kubernetes.io/projected/3ae333cb-5f6f-437f-9ba6-5cb249330692-kube-api-access-5qhl2\") pod \"kuadrant-operator-controller-manager-678d87766f-qx2hg\" (UID: \"3ae333cb-5f6f-437f-9ba6-5cb249330692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.160534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.160398 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae333cb-5f6f-437f-9ba6-5cb249330692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qx2hg\" (UID: \"3ae333cb-5f6f-437f-9ba6-5cb249330692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.261521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.261482 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qhl2\" (UniqueName: \"kubernetes.io/projected/3ae333cb-5f6f-437f-9ba6-5cb249330692-kube-api-access-5qhl2\") pod \"kuadrant-operator-controller-manager-678d87766f-qx2hg\" (UID: \"3ae333cb-5f6f-437f-9ba6-5cb249330692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.261745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.261581 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae333cb-5f6f-437f-9ba6-5cb249330692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qx2hg\" (UID: \"3ae333cb-5f6f-437f-9ba6-5cb249330692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.262075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.261958 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/3ae333cb-5f6f-437f-9ba6-5cb249330692-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-qx2hg\" (UID: \"3ae333cb-5f6f-437f-9ba6-5cb249330692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.319133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.319102 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qhl2\" (UniqueName: \"kubernetes.io/projected/3ae333cb-5f6f-437f-9ba6-5cb249330692-kube-api-access-5qhl2\") pod \"kuadrant-operator-controller-manager-678d87766f-qx2hg\" (UID: \"3ae333cb-5f6f-437f-9ba6-5cb249330692\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:07.608826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:07.608725 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:08.084612 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:08.084588 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg"] Nov 03 01:44:08.086215 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:44:08.086185 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae333cb_5f6f_437f_9ba6_5cb249330692.slice/crio-96f890041fca7d8ab4342af7b61dd8896f6a03a9e43a7658aaa7ad41fb925ba5 WatchSource:0}: Error finding container 96f890041fca7d8ab4342af7b61dd8896f6a03a9e43a7658aaa7ad41fb925ba5: Status 404 returned error can't find the container with id 96f890041fca7d8ab4342af7b61dd8896f6a03a9e43a7658aaa7ad41fb925ba5 Nov 03 01:44:08.307237 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:08.307197 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" event={"ID":"3ae333cb-5f6f-437f-9ba6-5cb249330692","Type":"ContainerStarted","Data":"b50834e4549381d845154974b0daaedad3e0a97ddd46435cb884dfdb1241419b"} Nov 03 01:44:08.307425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:08.307250 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" event={"ID":"3ae333cb-5f6f-437f-9ba6-5cb249330692","Type":"ContainerStarted","Data":"96f890041fca7d8ab4342af7b61dd8896f6a03a9e43a7658aaa7ad41fb925ba5"} Nov 03 01:44:08.307425 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:08.307413 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:08.424621 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:08.424516 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" podStartSLOduration=2.424502929 podStartE2EDuration="2.424502929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:08.422544976 +0000 UTC m=+5938.568239454" watchObservedRunningTime="2026-11-03 01:44:08.424502929 +0000 UTC m=+5938.570197384" Nov 03 01:44:12.156159 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:44:12.156116 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:44:15.027236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:15.027205 5144 scope.go:122] "RemoveContainer" containerID="8cc7518a327221b3d1fd53e6c941b52df16d8b27ea466f00c8061fbf8e29b57d" Nov 03 01:44:15.039102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:15.039071 5144 scope.go:122] "RemoveContainer" containerID="a080127e8b3ac77dda3360764f68081e601a263d3ecda224463144d08c05d671" Nov 03 01:44:15.048232 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:15.048213 5144 scope.go:122] "RemoveContainer" containerID="cfeb1975bf6107e55c34b5638f80342f4ac0c8339405d8dc76bbae261a987b22" Nov 03 01:44:19.316429 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:19.316394 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-qx2hg" Nov 03 01:44:35.135894 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.135858 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2"] Nov 03 01:44:35.139677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.139658 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.195578 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.195537 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2"] Nov 03 01:44:35.240941 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.240900 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:44:35.241165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.241040 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:44:35.262389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.262336 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.262604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.262516 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.262604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.262552 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-479x5\" (UniqueName: \"kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.363225 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.363179 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.363451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.363229 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-479x5\" (UniqueName: \"kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.363451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.363305 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.363902 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.363877 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.365875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.365856 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.419855 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.419786 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-479x5\" (UniqueName: \"kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5\") pod \"maas-api-revoke-sub-d576e68b-5vjz2\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.455837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.455801 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:35.835279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:35.835248 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2"] Nov 03 01:44:35.836702 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:44:35.836661 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9f1fca8_2189_4aed_81a6_d16512f882de.slice/crio-0658b1d059cc3cb51a8190ae2432a3e3fd6b297d96edc23c1cb06e08a4eb7d70 WatchSource:0}: Error finding container 0658b1d059cc3cb51a8190ae2432a3e3fd6b297d96edc23c1cb06e08a4eb7d70: Status 404 returned error can't find the container with id 0658b1d059cc3cb51a8190ae2432a3e3fd6b297d96edc23c1cb06e08a4eb7d70 Nov 03 01:44:36.492419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:36.492322 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" event={"ID":"a9f1fca8-2189-4aed-81a6-d16512f882de","Type":"ContainerStarted","Data":"1ee4da98d2365356d48a3465fb450dbe234f55fbb0ee642c3e00309317bdeb0a"} Nov 03 01:44:36.492419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:36.492374 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" event={"ID":"a9f1fca8-2189-4aed-81a6-d16512f882de","Type":"ContainerStarted","Data":"0658b1d059cc3cb51a8190ae2432a3e3fd6b297d96edc23c1cb06e08a4eb7d70"} Nov 03 01:44:36.604291 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:36.604229 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" podStartSLOduration=1.604207473 podStartE2EDuration="1.604207473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:36.603702679 +0000 UTC m=+5966.749397136" watchObservedRunningTime="2026-11-03 01:44:36.604207473 +0000 UTC m=+5966.749901929" Nov 03 01:44:37.500915 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:37.500882 5144 generic.go:356] "Generic (PLEG): container finished" podID="a9f1fca8-2189-4aed-81a6-d16512f882de" containerID="1ee4da98d2365356d48a3465fb450dbe234f55fbb0ee642c3e00309317bdeb0a" exitCode=0 Nov 03 01:44:37.501333 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:37.500972 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" event={"ID":"a9f1fca8-2189-4aed-81a6-d16512f882de","Type":"ContainerDied","Data":"1ee4da98d2365356d48a3465fb450dbe234f55fbb0ee642c3e00309317bdeb0a"} Nov 03 01:44:38.643369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.643344 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:38.698082 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.698048 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca\") pod \"a9f1fca8-2189-4aed-81a6-d16512f882de\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " Nov 03 01:44:38.698290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.698111 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca\") pod \"a9f1fca8-2189-4aed-81a6-d16512f882de\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " Nov 03 01:44:38.698290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.698268 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5\" (UniqueName: \"kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5\") pod \"a9f1fca8-2189-4aed-81a6-d16512f882de\" (UID: \"a9f1fca8-2189-4aed-81a6-d16512f882de\") " Nov 03 01:44:38.698560 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.698528 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca" pod "a9f1fca8-2189-4aed-81a6-d16512f882de" (UID: "a9f1fca8-2189-4aed-81a6-d16512f882de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:38.700412 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.700388 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca" pod "a9f1fca8-2189-4aed-81a6-d16512f882de" (UID: "a9f1fca8-2189-4aed-81a6-d16512f882de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:38.700522 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.700491 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5" pod "a9f1fca8-2189-4aed-81a6-d16512f882de" (UID: "a9f1fca8-2189-4aed-81a6-d16512f882de"). InnerVolumeSpecName "kube-api-access-479x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:38.799236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.799145 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-479x5\" (UniqueName: \"kubernetes.io/projected/a9f1fca8-2189-4aed-81a6-d16512f882de-kube-api-access-479x5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:44:38.799236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.799178 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:44:38.799236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:38.799193 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9f1fca8-2189-4aed-81a6-d16512f882de-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:44:39.514752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:39.514720 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" Nov 03 01:44:39.514976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:39.514758 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2" event={"ID":"a9f1fca8-2189-4aed-81a6-d16512f882de","Type":"ContainerDied","Data":"0658b1d059cc3cb51a8190ae2432a3e3fd6b297d96edc23c1cb06e08a4eb7d70"} Nov 03 01:44:39.514976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:39.514804 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0658b1d059cc3cb51a8190ae2432a3e3fd6b297d96edc23c1cb06e08a4eb7d70" Nov 03 01:44:42.156452 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:44:42.156415 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:44:47.046260 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:47.046209 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q"] Nov 03 01:44:47.095590 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:47.095553 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-rlv2q"] Nov 03 01:44:47.995091 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:47.995053 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06f71019-b864-42c9-935a-69223e059880" path="/var/lib/kubelet/pods/06f71019-b864-42c9-935a-69223e059880/volumes" Nov 03 01:44:50.033827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:50.033775 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:44:50.034357 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:50.033866 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:44:56.601974 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.601937 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr"] Nov 03 01:44:56.602416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.602387 5144 cpu_manager.go:414] "containerMap: removing container" podUID="a9f1fca8-2189-4aed-81a6-d16512f882de" containerName="revoke-keys" Nov 03 01:44:56.602416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.602398 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9f1fca8-2189-4aed-81a6-d16512f882de" containerName="revoke-keys" Nov 03 01:44:56.602507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.602497 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9f1fca8-2189-4aed-81a6-d16512f882de" containerName="revoke-keys" Nov 03 01:44:56.606058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.606041 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.662269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.662239 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr"] Nov 03 01:44:56.708570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.708530 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:44:56.708798 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.708692 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:44:56.719400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.719358 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.719646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.719519 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.719646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.719621 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t22hf\" (UniqueName: \"kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.820703 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.820661 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.820907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.820771 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t22hf\" (UniqueName: \"kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.820907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.820840 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.821362 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.821339 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.823415 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.823389 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.877459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.877378 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t22hf\" (UniqueName: \"kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf\") pod \"maas-api-revoke-sub-d2bc93bc-nqrhr\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:56.919308 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:56.919268 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:57.294033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:57.293986 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr"] Nov 03 01:44:57.295729 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:44:57.295701 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf426b483_f5c1_4dc7_afcd_7118d77468fc.slice/crio-7ba7debf4f71f11b2e00ad163e431512d4849f0716e90cf7302566b06ac71545 WatchSource:0}: Error finding container 7ba7debf4f71f11b2e00ad163e431512d4849f0716e90cf7302566b06ac71545: Status 404 returned error can't find the container with id 7ba7debf4f71f11b2e00ad163e431512d4849f0716e90cf7302566b06ac71545 Nov 03 01:44:57.694386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:57.694346 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" event={"ID":"f426b483-f5c1-4dc7-afcd-7118d77468fc","Type":"ContainerStarted","Data":"2b60e6f6a1b96efca74a7144dfae6758803b830aad043bf37baaf389f4f1ca36"} Nov 03 01:44:57.694386 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:57.694404 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" event={"ID":"f426b483-f5c1-4dc7-afcd-7118d77468fc","Type":"ContainerStarted","Data":"7ba7debf4f71f11b2e00ad163e431512d4849f0716e90cf7302566b06ac71545"} Nov 03 01:44:57.807520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:57.807452 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" podStartSLOduration=1.807436343 podStartE2EDuration="1.807436343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:44:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:44:57.806237115 +0000 UTC m=+5987.951931570" watchObservedRunningTime="2026-11-03 01:44:57.807436343 +0000 UTC m=+5987.953130797" Nov 03 01:44:58.701848 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:58.701811 5144 generic.go:356] "Generic (PLEG): container finished" podID="f426b483-f5c1-4dc7-afcd-7118d77468fc" containerID="2b60e6f6a1b96efca74a7144dfae6758803b830aad043bf37baaf389f4f1ca36" exitCode=0 Nov 03 01:44:58.702328 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:58.701896 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" event={"ID":"f426b483-f5c1-4dc7-afcd-7118d77468fc","Type":"ContainerDied","Data":"2b60e6f6a1b96efca74a7144dfae6758803b830aad043bf37baaf389f4f1ca36"} Nov 03 01:44:59.051916 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.051880 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx"] Nov 03 01:44:59.057142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.057114 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-fklsx"] Nov 03 01:44:59.852476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.852442 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:44:59.954209 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.954166 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca\") pod \"f426b483-f5c1-4dc7-afcd-7118d77468fc\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " Nov 03 01:44:59.954403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.954253 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf\" (UniqueName: \"kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf\") pod \"f426b483-f5c1-4dc7-afcd-7118d77468fc\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " Nov 03 01:44:59.954403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.954366 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca\") pod \"f426b483-f5c1-4dc7-afcd-7118d77468fc\" (UID: \"f426b483-f5c1-4dc7-afcd-7118d77468fc\") " Nov 03 01:44:59.954582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.954531 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca" pod "f426b483-f5c1-4dc7-afcd-7118d77468fc" (UID: "f426b483-f5c1-4dc7-afcd-7118d77468fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:44:59.954812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.954789 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:44:59.956596 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.956567 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca" pod "f426b483-f5c1-4dc7-afcd-7118d77468fc" (UID: "f426b483-f5c1-4dc7-afcd-7118d77468fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:44:59.956723 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.956674 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf" pod "f426b483-f5c1-4dc7-afcd-7118d77468fc" (UID: "f426b483-f5c1-4dc7-afcd-7118d77468fc"). InnerVolumeSpecName "kube-api-access-t22hf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:44:59.995465 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:44:59.995432 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c1a557-b170-41dc-b76c-a204ad705b33" path="/var/lib/kubelet/pods/a3c1a557-b170-41dc-b76c-a204ad705b33/volumes" Nov 03 01:45:00.055444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.055404 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t22hf\" (UniqueName: \"kubernetes.io/projected/f426b483-f5c1-4dc7-afcd-7118d77468fc-kube-api-access-t22hf\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:00.055444 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.055439 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f426b483-f5c1-4dc7-afcd-7118d77468fc-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:00.152089 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.152046 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk"] Nov 03 01:45:00.152602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.152581 5144 cpu_manager.go:414] "containerMap: removing container" podUID="f426b483-f5c1-4dc7-afcd-7118d77468fc" containerName="revoke-keys" Nov 03 01:45:00.152729 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.152604 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f426b483-f5c1-4dc7-afcd-7118d77468fc" containerName="revoke-keys" Nov 03 01:45:00.152826 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.152810 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="f426b483-f5c1-4dc7-afcd-7118d77468fc" containerName="revoke-keys" Nov 03 01:45:00.156151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.156131 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.212215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.212180 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk"] Nov 03 01:45:00.257825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.257789 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-xdfvk\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.258078 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.257987 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh2p8\" (UniqueName: \"kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8\") pod \"maas-api-key-cleanup-29894505-xdfvk\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.359068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.359006 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh2p8\" (UniqueName: \"kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8\") pod \"maas-api-key-cleanup-29894505-xdfvk\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.359068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.359073 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-xdfvk\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.359711 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.359692 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894505-xdfvk\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.416495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.416459 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh2p8\" (UniqueName: \"kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8\") pod \"maas-api-key-cleanup-29894505-xdfvk\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.471814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.471775 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:00.715194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.715158 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" event={"ID":"f426b483-f5c1-4dc7-afcd-7118d77468fc","Type":"ContainerDied","Data":"7ba7debf4f71f11b2e00ad163e431512d4849f0716e90cf7302566b06ac71545"} Nov 03 01:45:00.715420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.715219 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ba7debf4f71f11b2e00ad163e431512d4849f0716e90cf7302566b06ac71545" Nov 03 01:45:00.715420 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.715176 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr" Nov 03 01:45:00.847025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:00.847000 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk"] Nov 03 01:45:00.848348 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:45:00.848312 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7f5ebeb_a1cf_4805_a428_b69391c522d6.slice/crio-b7044f09cdeffd542983c455cebe35619b60ae4aa96dcedaf80f8b5d12db911a WatchSource:0}: Error finding container b7044f09cdeffd542983c455cebe35619b60ae4aa96dcedaf80f8b5d12db911a: Status 404 returned error can't find the container with id b7044f09cdeffd542983c455cebe35619b60ae4aa96dcedaf80f8b5d12db911a Nov 03 01:45:01.722930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:01.722890 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerStarted","Data":"ef619aa6b81c65cff4f6e29b3789e5aaf9a1496b3f8493a34d674515be2af1ee"} Nov 03 01:45:01.723343 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:01.722954 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerStarted","Data":"b7044f09cdeffd542983c455cebe35619b60ae4aa96dcedaf80f8b5d12db911a"} Nov 03 01:45:01.839738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:01.839686 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" podStartSLOduration=1.839671149 podStartE2EDuration="1.839671149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:01.837267661 +0000 UTC m=+5991.982962118" watchObservedRunningTime="2026-11-03 01:45:01.839671149 +0000 UTC m=+5991.985365606" Nov 03 01:45:02.730419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:02.730383 5144 generic.go:356] "Generic (PLEG): container finished" podID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerID="ef619aa6b81c65cff4f6e29b3789e5aaf9a1496b3f8493a34d674515be2af1ee" exitCode=60 Nov 03 01:45:02.730881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:02.730469 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerDied","Data":"ef619aa6b81c65cff4f6e29b3789e5aaf9a1496b3f8493a34d674515be2af1ee"} Nov 03 01:45:02.730881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:02.730695 5144 scope.go:122] "RemoveContainer" containerID="ef619aa6b81c65cff4f6e29b3789e5aaf9a1496b3f8493a34d674515be2af1ee" Nov 03 01:45:03.738646 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:03.738600 5144 generic.go:356] "Generic (PLEG): container finished" podID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerID="6a481e6e48be1eba6c3e93cbd9e8e9290691f874d4f4e175ec063d8f7e37da85" exitCode=60 Nov 03 01:45:03.739132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:03.738700 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerDied","Data":"6a481e6e48be1eba6c3e93cbd9e8e9290691f874d4f4e175ec063d8f7e37da85"} Nov 03 01:45:03.739132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:03.738746 5144 scope.go:122] "RemoveContainer" containerID="ef619aa6b81c65cff4f6e29b3789e5aaf9a1496b3f8493a34d674515be2af1ee" Nov 03 01:45:03.739132 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:03.739012 5144 scope.go:122] "RemoveContainer" containerID="6a481e6e48be1eba6c3e93cbd9e8e9290691f874d4f4e175ec063d8f7e37da85" Nov 03 01:45:03.739324 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:45:03.739303 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-xdfvk_odh-ai-gateway-infra(c7f5ebeb-a1cf-4805-a428-b69391c522d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" Nov 03 01:45:04.759335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:04.759304 5144 scope.go:122] "RemoveContainer" containerID="6a481e6e48be1eba6c3e93cbd9e8e9290691f874d4f4e175ec063d8f7e37da85" Nov 03 01:45:04.759847 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:45:04.759574 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-xdfvk_odh-ai-gateway-infra(c7f5ebeb-a1cf-4805-a428-b69391c522d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" Nov 03 01:45:11.385846 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:11.385811 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:45:11.396288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:11.396261 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:45:12.157337 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:45:12.157299 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:45:13.989119 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:13.989083 5144 scope.go:122] "RemoveContainer" containerID="6a481e6e48be1eba6c3e93cbd9e8e9290691f874d4f4e175ec063d8f7e37da85" Nov 03 01:45:14.822681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:14.822621 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerStarted","Data":"9121bca746847837bcd81e5e5a67d18a149a00526ffcbe1d971e182c29ab2675"} Nov 03 01:45:15.077813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.077727 5144 scope.go:122] "RemoveContainer" containerID="2d4e4b2e3f97446d04b86b78fbfca3552f13aadc5dd42904f2b8bdaa528e489d" Nov 03 01:45:15.089027 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.089005 5144 scope.go:122] "RemoveContainer" containerID="bfb4d21d56f09323e9c066ae41418562d1d331d090aad54ffb60166e5946142b" Nov 03 01:45:15.098273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.098254 5144 scope.go:122] "RemoveContainer" containerID="513b6e6a5287af4ab3302dd9b59fcface9d87fff0ed2cea7e3238e0ceb0e0196" Nov 03 01:45:15.830924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.830889 5144 generic.go:356] "Generic (PLEG): container finished" podID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerID="9121bca746847837bcd81e5e5a67d18a149a00526ffcbe1d971e182c29ab2675" exitCode=60 Nov 03 01:45:15.831114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.830958 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerDied","Data":"9121bca746847837bcd81e5e5a67d18a149a00526ffcbe1d971e182c29ab2675"} Nov 03 01:45:15.831114 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.831053 5144 scope.go:122] "RemoveContainer" containerID="6a481e6e48be1eba6c3e93cbd9e8e9290691f874d4f4e175ec063d8f7e37da85" Nov 03 01:45:15.831405 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.831389 5144 scope.go:122] "RemoveContainer" containerID="9121bca746847837bcd81e5e5a67d18a149a00526ffcbe1d971e182c29ab2675" Nov 03 01:45:15.831741 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:45:15.831715 5144 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894505-xdfvk_odh-ai-gateway-infra(c7f5ebeb-a1cf-4805-a428-b69391c522d6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" Nov 03 01:45:15.943847 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:15.943808 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk"] Nov 03 01:45:16.982480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:16.982447 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:17.134339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.134228 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8\" (UniqueName: \"kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8\") pod \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " Nov 03 01:45:17.134339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.134288 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca\") pod \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\" (UID: \"c7f5ebeb-a1cf-4805-a428-b69391c522d6\") " Nov 03 01:45:17.134686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.134623 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca" pod "c7f5ebeb-a1cf-4805-a428-b69391c522d6" (UID: "c7f5ebeb-a1cf-4805-a428-b69391c522d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:17.135033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.135003 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7f5ebeb-a1cf-4805-a428-b69391c522d6-maas-api-service-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:17.136860 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.136830 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8" pod "c7f5ebeb-a1cf-4805-a428-b69391c522d6" (UID: "c7f5ebeb-a1cf-4805-a428-b69391c522d6"). InnerVolumeSpecName "kube-api-access-gh2p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:17.236254 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.236198 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh2p8\" (UniqueName: \"kubernetes.io/projected/c7f5ebeb-a1cf-4805-a428-b69391c522d6-kube-api-access-gh2p8\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:17.846426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.846391 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" Nov 03 01:45:17.846426 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.846407 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk" event={"ID":"c7f5ebeb-a1cf-4805-a428-b69391c522d6","Type":"ContainerDied","Data":"b7044f09cdeffd542983c455cebe35619b60ae4aa96dcedaf80f8b5d12db911a"} Nov 03 01:45:17.846698 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:17.846471 5144 scope.go:122] "RemoveContainer" containerID="9121bca746847837bcd81e5e5a67d18a149a00526ffcbe1d971e182c29ab2675" Nov 03 01:45:18.008672 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:18.008624 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk"] Nov 03 01:45:18.058093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:18.058060 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894505-xdfvk"] Nov 03 01:45:19.994888 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:19.994856 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" path="/var/lib/kubelet/pods/c7f5ebeb-a1cf-4805-a428-b69391c522d6/volumes" Nov 03 01:45:33.872896 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.872841 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn"] Nov 03 01:45:33.873513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873490 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873516 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873531 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873538 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873554 5144 cpu_manager.go:414] "containerMap: removing container" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873617 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873562 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873943 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873728 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873943 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873742 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.873943 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.873754 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7f5ebeb-a1cf-4805-a428-b69391c522d6" containerName="cleanup" Nov 03 01:45:33.877075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.877053 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:33.932587 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.932550 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn"] Nov 03 01:45:33.952125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.952086 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:33.952125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.952134 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:33.952419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.952241 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcltr\" (UniqueName: \"kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:33.978821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.978787 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:45:33.979007 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:33.978889 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:45:34.054688 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.054579 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.055010 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.054969 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.055167 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.055153 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcltr\" (UniqueName: \"kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.056463 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.056433 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.062979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.059712 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.111739 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.111702 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcltr\" (UniqueName: \"kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr\") pod \"maas-api-revoke-sub-2b39b932-fc5qn\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.189554 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.189451 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:34.563667 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:34.563623 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn"] Nov 03 01:45:34.566021 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:45:34.565987 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880843af_cb33_4198_b699_22fd72d9b133.slice/crio-28465fa7b66d591cdfd57c0c4ece400a120c71c4c29291a7b22ec311b9e29edc WatchSource:0}: Error finding container 28465fa7b66d591cdfd57c0c4ece400a120c71c4c29291a7b22ec311b9e29edc: Status 404 returned error can't find the container with id 28465fa7b66d591cdfd57c0c4ece400a120c71c4c29291a7b22ec311b9e29edc Nov 03 01:45:35.013366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:35.013328 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" event={"ID":"880843af-cb33-4198-b699-22fd72d9b133","Type":"ContainerStarted","Data":"31cb745c302e28b645b793f9c4a125b5b8e8a2a307261a367fd6f55f7786fb7a"} Nov 03 01:45:35.013815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:35.013417 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" event={"ID":"880843af-cb33-4198-b699-22fd72d9b133","Type":"ContainerStarted","Data":"28465fa7b66d591cdfd57c0c4ece400a120c71c4c29291a7b22ec311b9e29edc"} Nov 03 01:45:35.129176 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:35.129071 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" podStartSLOduration=2.129056613 podStartE2EDuration="2.129056613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:35.12770894 +0000 UTC m=+6025.273403420" watchObservedRunningTime="2026-11-03 01:45:35.129056613 +0000 UTC m=+6025.274751121" Nov 03 01:45:36.020973 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:36.020931 5144 generic.go:356] "Generic (PLEG): container finished" podID="880843af-cb33-4198-b699-22fd72d9b133" containerID="31cb745c302e28b645b793f9c4a125b5b8e8a2a307261a367fd6f55f7786fb7a" exitCode=0 Nov 03 01:45:36.021419 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:36.021007 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" event={"ID":"880843af-cb33-4198-b699-22fd72d9b133","Type":"ContainerDied","Data":"31cb745c302e28b645b793f9c4a125b5b8e8a2a307261a367fd6f55f7786fb7a"} Nov 03 01:45:37.173613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.173583 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:37.287312 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.287205 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca\") pod \"880843af-cb33-4198-b699-22fd72d9b133\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " Nov 03 01:45:37.287519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.287331 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca\") pod \"880843af-cb33-4198-b699-22fd72d9b133\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " Nov 03 01:45:37.287519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.287379 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr\" (UniqueName: \"kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr\") pod \"880843af-cb33-4198-b699-22fd72d9b133\" (UID: \"880843af-cb33-4198-b699-22fd72d9b133\") " Nov 03 01:45:37.287747 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.287724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca" pod "880843af-cb33-4198-b699-22fd72d9b133" (UID: "880843af-cb33-4198-b699-22fd72d9b133"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:37.290335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.290304 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr" pod "880843af-cb33-4198-b699-22fd72d9b133" (UID: "880843af-cb33-4198-b699-22fd72d9b133"). InnerVolumeSpecName "kube-api-access-zcltr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:37.290593 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.290573 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca" pod "880843af-cb33-4198-b699-22fd72d9b133" (UID: "880843af-cb33-4198-b699-22fd72d9b133"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:37.388373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.388331 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/880843af-cb33-4198-b699-22fd72d9b133-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:37.388373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.388368 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcltr\" (UniqueName: \"kubernetes.io/projected/880843af-cb33-4198-b699-22fd72d9b133-kube-api-access-zcltr\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:37.388373 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:37.388378 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/880843af-cb33-4198-b699-22fd72d9b133-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:38.033494 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:38.033463 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" Nov 03 01:45:38.033671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:38.033458 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn" event={"ID":"880843af-cb33-4198-b699-22fd72d9b133","Type":"ContainerDied","Data":"28465fa7b66d591cdfd57c0c4ece400a120c71c4c29291a7b22ec311b9e29edc"} Nov 03 01:45:38.033671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:38.033603 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28465fa7b66d591cdfd57c0c4ece400a120c71c4c29291a7b22ec311b9e29edc" Nov 03 01:45:42.155207 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:45:42.155167 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:45:44.626277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.626239 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6"] Nov 03 01:45:44.626975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.626882 5144 cpu_manager.go:414] "containerMap: removing container" podUID="880843af-cb33-4198-b699-22fd72d9b133" containerName="revoke-keys" Nov 03 01:45:44.626975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.626900 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="880843af-cb33-4198-b699-22fd72d9b133" containerName="revoke-keys" Nov 03 01:45:44.626975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.626975 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="880843af-cb33-4198-b699-22fd72d9b133" containerName="revoke-keys" Nov 03 01:45:44.630133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.630116 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.682583 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.682530 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:45:44.731987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.731958 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:45:44.734824 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.734798 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6"] Nov 03 01:45:44.769220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.769181 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.769220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.769227 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7szw2\" (UniqueName: \"kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.769453 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.769249 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.870326 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.870287 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.870523 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.870335 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7szw2\" (UniqueName: \"kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.870523 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.870359 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.870965 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.870942 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.872908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.872888 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.927582 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.927504 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7szw2\" (UniqueName: \"kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2\") pod \"maas-api-revoke-sub-cc82e4a3-2w4s6\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:44.943241 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:44.943206 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:45.315124 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:45.315097 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6"] Nov 03 01:45:45.317115 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:45:45.317084 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod655f05f4_ebf1_4cca_bcb3_39fcd65d94d9.slice/crio-69e0e0e071756063f941d5ce9ec737a7527de3f4b5dea9a7ea1e34a0a64c600b WatchSource:0}: Error finding container 69e0e0e071756063f941d5ce9ec737a7527de3f4b5dea9a7ea1e34a0a64c600b: Status 404 returned error can't find the container with id 69e0e0e071756063f941d5ce9ec737a7527de3f4b5dea9a7ea1e34a0a64c600b Nov 03 01:45:46.080012 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:46.079972 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" event={"ID":"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9","Type":"ContainerStarted","Data":"182f95d57edf821804eedc6e2744210d331dfb2e2ae9efc29b45343cc48db27d"} Nov 03 01:45:46.080427 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:46.080030 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" event={"ID":"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9","Type":"ContainerStarted","Data":"69e0e0e071756063f941d5ce9ec737a7527de3f4b5dea9a7ea1e34a0a64c600b"} Nov 03 01:45:46.193477 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:46.193294 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" podStartSLOduration=2.193281105 podStartE2EDuration="2.193281105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:45:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:45:46.19177465 +0000 UTC m=+6036.337469116" watchObservedRunningTime="2026-11-03 01:45:46.193281105 +0000 UTC m=+6036.338975582" Nov 03 01:45:47.087408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:47.087369 5144 generic.go:356] "Generic (PLEG): container finished" podID="655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" containerID="182f95d57edf821804eedc6e2744210d331dfb2e2ae9efc29b45343cc48db27d" exitCode=0 Nov 03 01:45:47.087865 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:47.087438 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" event={"ID":"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9","Type":"ContainerDied","Data":"182f95d57edf821804eedc6e2744210d331dfb2e2ae9efc29b45343cc48db27d"} Nov 03 01:45:48.239805 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.239770 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:48.405238 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.405141 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca\") pod \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " Nov 03 01:45:48.405432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.405241 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2\" (UniqueName: \"kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2\") pod \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " Nov 03 01:45:48.405432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.405300 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca\") pod \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\" (UID: \"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9\") " Nov 03 01:45:48.405599 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.405574 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca" pod "655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" (UID: "655f05f4-ebf1-4cca-bcb3-39fcd65d94d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:45:48.407478 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.407451 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca" pod "655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" (UID: "655f05f4-ebf1-4cca-bcb3-39fcd65d94d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:45:48.407572 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.407505 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2" pod "655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" (UID: "655f05f4-ebf1-4cca-bcb3-39fcd65d94d9"). InnerVolumeSpecName "kube-api-access-7szw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:45:48.506726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.506688 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:48.506726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.506720 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:48.506726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:48.506733 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7szw2\" (UniqueName: \"kubernetes.io/projected/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9-kube-api-access-7szw2\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:45:49.099567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:49.099539 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" Nov 03 01:45:49.099770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:49.099537 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6" event={"ID":"655f05f4-ebf1-4cca-bcb3-39fcd65d94d9","Type":"ContainerDied","Data":"69e0e0e071756063f941d5ce9ec737a7527de3f4b5dea9a7ea1e34a0a64c600b"} Nov 03 01:45:49.099770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:45:49.099680 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="69e0e0e071756063f941d5ce9ec737a7527de3f4b5dea9a7ea1e34a0a64c600b" Nov 03 01:46:12.156387 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:46:12.156346 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:46:15.135734 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:15.135704 5144 scope.go:122] "RemoveContainer" containerID="f1a201e17db81cbbd08e14d9ec7b72a86f60be9e3c735f35e2cb116ef6b465ac" Nov 03 01:46:15.144908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:15.144889 5144 scope.go:122] "RemoveContainer" containerID="5bba89ab32ed1a1dc7d587d11fc862214788a3c475072c90abb86f919075c424" Nov 03 01:46:15.153927 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:15.153905 5144 scope.go:122] "RemoveContainer" containerID="218478d41708b109afb729c0e6e8ffa21335a1672040c415a8e26fcdcfc4a3fd" Nov 03 01:46:20.256349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.256273 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47"] Nov 03 01:46:20.256875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.256767 5144 cpu_manager.go:414] "containerMap: removing container" podUID="655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" containerName="revoke-keys" Nov 03 01:46:20.256875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.256779 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" containerName="revoke-keys" Nov 03 01:46:20.256968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.256879 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" containerName="revoke-keys" Nov 03 01:46:20.261776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.261756 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.314128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.314100 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.314316 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.314247 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxw7q\" (UniqueName: \"kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.314404 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.314330 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.315518 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.315493 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47"] Nov 03 01:46:20.361647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.361610 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:46:20.361838 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.361821 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:46:20.415054 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.415027 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.415196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.415080 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxw7q\" (UniqueName: \"kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.415196 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.415115 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.415797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.415778 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.417542 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.417520 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.471530 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.471496 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxw7q\" (UniqueName: \"kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q\") pod \"maas-api-revoke-sub-669bbc7b-4mn47\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.578797 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.578734 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:20.952193 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:20.952169 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47"] Nov 03 01:46:20.954700 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:46:20.954675 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ca6874_189d_4ab7_8421_0ae0122caed3.slice/crio-cc59e5cd25ac4b144d5e86e7aef153010608ba92fb91a52d7af85b3caeba16e6 WatchSource:0}: Error finding container cc59e5cd25ac4b144d5e86e7aef153010608ba92fb91a52d7af85b3caeba16e6: Status 404 returned error can't find the container with id cc59e5cd25ac4b144d5e86e7aef153010608ba92fb91a52d7af85b3caeba16e6 Nov 03 01:46:21.302874 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:21.302842 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" event={"ID":"68ca6874-189d-4ab7-8421-0ae0122caed3","Type":"ContainerStarted","Data":"d3034412f96d23480effe6f07658ba8e0fdc914cde3b361bef73e70208a24091"} Nov 03 01:46:21.303296 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:21.302902 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" event={"ID":"68ca6874-189d-4ab7-8421-0ae0122caed3","Type":"ContainerStarted","Data":"cc59e5cd25ac4b144d5e86e7aef153010608ba92fb91a52d7af85b3caeba16e6"} Nov 03 01:46:21.418215 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:21.417857 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" podStartSLOduration=1.417839384 podStartE2EDuration="1.417839384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:46:21.417585748 +0000 UTC m=+6071.563280215" watchObservedRunningTime="2026-11-03 01:46:21.417839384 +0000 UTC m=+6071.563533839" Nov 03 01:46:22.309601 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:22.309562 5144 generic.go:356] "Generic (PLEG): container finished" podID="68ca6874-189d-4ab7-8421-0ae0122caed3" containerID="d3034412f96d23480effe6f07658ba8e0fdc914cde3b361bef73e70208a24091" exitCode=0 Nov 03 01:46:22.310048 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:22.309666 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" event={"ID":"68ca6874-189d-4ab7-8421-0ae0122caed3","Type":"ContainerDied","Data":"d3034412f96d23480effe6f07658ba8e0fdc914cde3b361bef73e70208a24091"} Nov 03 01:46:23.447400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.447374 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:23.548842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.548813 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca\") pod \"68ca6874-189d-4ab7-8421-0ae0122caed3\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " Nov 03 01:46:23.548996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.548891 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q\" (UniqueName: \"kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q\") pod \"68ca6874-189d-4ab7-8421-0ae0122caed3\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " Nov 03 01:46:23.548996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.548931 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca\") pod \"68ca6874-189d-4ab7-8421-0ae0122caed3\" (UID: \"68ca6874-189d-4ab7-8421-0ae0122caed3\") " Nov 03 01:46:23.549184 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.549163 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca" pod "68ca6874-189d-4ab7-8421-0ae0122caed3" (UID: "68ca6874-189d-4ab7-8421-0ae0122caed3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:46:23.549324 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.549303 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:46:23.551150 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.551129 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q" pod "68ca6874-189d-4ab7-8421-0ae0122caed3" (UID: "68ca6874-189d-4ab7-8421-0ae0122caed3"). InnerVolumeSpecName "kube-api-access-dxw7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:46:23.551234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.551154 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca" pod "68ca6874-189d-4ab7-8421-0ae0122caed3" (UID: "68ca6874-189d-4ab7-8421-0ae0122caed3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:46:23.650382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.650316 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxw7q\" (UniqueName: \"kubernetes.io/projected/68ca6874-189d-4ab7-8421-0ae0122caed3-kube-api-access-dxw7q\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:46:23.650382 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:23.650351 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68ca6874-189d-4ab7-8421-0ae0122caed3-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:46:24.321099 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:24.321066 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" Nov 03 01:46:24.321287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:24.321071 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47" event={"ID":"68ca6874-189d-4ab7-8421-0ae0122caed3","Type":"ContainerDied","Data":"cc59e5cd25ac4b144d5e86e7aef153010608ba92fb91a52d7af85b3caeba16e6"} Nov 03 01:46:24.321287 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:24.321191 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc59e5cd25ac4b144d5e86e7aef153010608ba92fb91a52d7af85b3caeba16e6" Nov 03 01:46:42.155715 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:46:42.155674 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:46:51.045314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:51.045278 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42"] Nov 03 01:46:51.094781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:51.094752 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-adbe0b63-hqw42"] Nov 03 01:46:51.995476 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:46:51.995439 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78800654-60c1-4906-b3e8-206f615c11c6" path="/var/lib/kubelet/pods/78800654-60c1-4906-b3e8-206f615c11c6/volumes" Nov 03 01:47:05.080707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.080665 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525"] Nov 03 01:47:05.081128 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.081116 5144 cpu_manager.go:414] "containerMap: removing container" podUID="68ca6874-189d-4ab7-8421-0ae0122caed3" containerName="revoke-keys" Nov 03 01:47:05.081191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.081130 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68ca6874-189d-4ab7-8421-0ae0122caed3" containerName="revoke-keys" Nov 03 01:47:05.081236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.081218 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="68ca6874-189d-4ab7-8421-0ae0122caed3" containerName="revoke-keys" Nov 03 01:47:05.084360 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.084340 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.142323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.142286 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525"] Nov 03 01:47:05.156850 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.156810 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.157024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.156869 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.157024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.156972 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrrdh\" (UniqueName: \"kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.188276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.188236 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:47:05.188456 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.188314 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:47:05.258093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.258056 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.258093 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.258098 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.258335 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.258133 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrrdh\" (UniqueName: \"kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.258794 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.258768 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.260774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.260744 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.314947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.314911 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrrdh\" (UniqueName: \"kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh\") pod \"maas-api-revoke-sub-77111d9f-m6525\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.398966 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.398867 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:05.772616 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:05.772587 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525"] Nov 03 01:47:05.773969 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:47:05.773938 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1bc4a8_227f_408a_a2cb_de3b952622d5.slice/crio-b2ae4ed5d512d32a70785d2e3771a43173f752be3114f2e090c8adb7cdc2831f WatchSource:0}: Error finding container b2ae4ed5d512d32a70785d2e3771a43173f752be3114f2e090c8adb7cdc2831f: Status 404 returned error can't find the container with id b2ae4ed5d512d32a70785d2e3771a43173f752be3114f2e090c8adb7cdc2831f Nov 03 01:47:06.590907 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:06.590871 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" event={"ID":"eb1bc4a8-227f-408a-a2cb-de3b952622d5","Type":"ContainerStarted","Data":"2e555c6c35b48a7d61b9db6409006cb1f76ba2073ba0ac6714deef503578c6cf"} Nov 03 01:47:06.591389 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:06.590930 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" event={"ID":"eb1bc4a8-227f-408a-a2cb-de3b952622d5","Type":"ContainerStarted","Data":"b2ae4ed5d512d32a70785d2e3771a43173f752be3114f2e090c8adb7cdc2831f"} Nov 03 01:47:06.704101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:06.704046 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" podStartSLOduration=1.704030846 podStartE2EDuration="1.704030846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:47:06.702210935 +0000 UTC m=+6116.847905391" watchObservedRunningTime="2026-11-03 01:47:06.704030846 +0000 UTC m=+6116.849725302" Nov 03 01:47:07.600304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:07.600265 5144 generic.go:356] "Generic (PLEG): container finished" podID="eb1bc4a8-227f-408a-a2cb-de3b952622d5" containerID="2e555c6c35b48a7d61b9db6409006cb1f76ba2073ba0ac6714deef503578c6cf" exitCode=0 Nov 03 01:47:07.600774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:07.600354 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" event={"ID":"eb1bc4a8-227f-408a-a2cb-de3b952622d5","Type":"ContainerDied","Data":"2e555c6c35b48a7d61b9db6409006cb1f76ba2073ba0ac6714deef503578c6cf"} Nov 03 01:47:08.755148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.755103 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:08.898070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.897961 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh\" (UniqueName: \"kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh\") pod \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " Nov 03 01:47:08.898070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.898034 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca\") pod \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " Nov 03 01:47:08.898337 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.898084 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca\") pod \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\" (UID: \"eb1bc4a8-227f-408a-a2cb-de3b952622d5\") " Nov 03 01:47:08.898495 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.898466 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca" pod "eb1bc4a8-227f-408a-a2cb-de3b952622d5" (UID: "eb1bc4a8-227f-408a-a2cb-de3b952622d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:08.900400 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.900368 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca" pod "eb1bc4a8-227f-408a-a2cb-de3b952622d5" (UID: "eb1bc4a8-227f-408a-a2cb-de3b952622d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:08.900520 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.900490 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh" pod "eb1bc4a8-227f-408a-a2cb-de3b952622d5" (UID: "eb1bc4a8-227f-408a-a2cb-de3b952622d5"). InnerVolumeSpecName "kube-api-access-lrrdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:08.999710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.999622 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrrdh\" (UniqueName: \"kubernetes.io/projected/eb1bc4a8-227f-408a-a2cb-de3b952622d5-kube-api-access-lrrdh\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:47:08.999710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.999705 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:47:08.999710 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:08.999715 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb1bc4a8-227f-408a-a2cb-de3b952622d5-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:47:09.612945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:09.612911 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" Nov 03 01:47:09.612945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:09.612936 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525" event={"ID":"eb1bc4a8-227f-408a-a2cb-de3b952622d5","Type":"ContainerDied","Data":"b2ae4ed5d512d32a70785d2e3771a43173f752be3114f2e090c8adb7cdc2831f"} Nov 03 01:47:09.613178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:09.612987 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2ae4ed5d512d32a70785d2e3771a43173f752be3114f2e090c8adb7cdc2831f" Nov 03 01:47:12.154980 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:47:12.154945 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:47:15.876875 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.876838 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4"] Nov 03 01:47:15.877307 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.877293 5144 cpu_manager.go:414] "containerMap: removing container" podUID="eb1bc4a8-227f-408a-a2cb-de3b952622d5" containerName="revoke-keys" Nov 03 01:47:15.877354 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.877307 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb1bc4a8-227f-408a-a2cb-de3b952622d5" containerName="revoke-keys" Nov 03 01:47:15.877414 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.877405 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb1bc4a8-227f-408a-a2cb-de3b952622d5" containerName="revoke-keys" Nov 03 01:47:15.880775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.880756 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:15.937722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.937685 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4"] Nov 03 01:47:15.970925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.970883 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44vt6\" (UniqueName: \"kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:15.971122 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.970934 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:15.971122 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.971036 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:15.982812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.982770 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:47:15.982812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:15.982801 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:47:16.071972 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.071930 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44vt6\" (UniqueName: \"kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.072133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.071988 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.072133 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.072031 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.072770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.072750 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.074487 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.074459 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.128562 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.128484 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44vt6\" (UniqueName: \"kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6\") pod \"maas-api-revoke-sub-e3d58e9d-krvk4\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.198406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.198364 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:16.572993 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.572967 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4"] Nov 03 01:47:16.574850 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:47:16.574818 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09456d5f_e8fe_432a_b11f_093f94a8a8e7.slice/crio-9bada44aca188256d9490b975aec9ad03222deabe6bb4ec591d8612580023fae WatchSource:0}: Error finding container 9bada44aca188256d9490b975aec9ad03222deabe6bb4ec591d8612580023fae: Status 404 returned error can't find the container with id 9bada44aca188256d9490b975aec9ad03222deabe6bb4ec591d8612580023fae Nov 03 01:47:16.657056 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:16.657017 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" event={"ID":"09456d5f-e8fe-432a-b11f-093f94a8a8e7","Type":"ContainerStarted","Data":"9bada44aca188256d9490b975aec9ad03222deabe6bb4ec591d8612580023fae"} Nov 03 01:47:17.665052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:17.664943 5144 generic.go:356] "Generic (PLEG): container finished" podID="09456d5f-e8fe-432a-b11f-093f94a8a8e7" containerID="9c6e275d438cde62b57bf8cc76dcd54e75993108c6c6211f9c8afa2273111d1b" exitCode=0 Nov 03 01:47:17.665052 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:17.665037 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" event={"ID":"09456d5f-e8fe-432a-b11f-093f94a8a8e7","Type":"ContainerDied","Data":"9c6e275d438cde62b57bf8cc76dcd54e75993108c6c6211f9c8afa2273111d1b"} Nov 03 01:47:18.821111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.821079 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:18.900708 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.900671 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca\") pod \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " Nov 03 01:47:18.900924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.900745 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca\") pod \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " Nov 03 01:47:18.900924 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.900818 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6\" (UniqueName: \"kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6\") pod \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\" (UID: \"09456d5f-e8fe-432a-b11f-093f94a8a8e7\") " Nov 03 01:47:18.901129 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.901108 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca" pod "09456d5f-e8fe-432a-b11f-093f94a8a8e7" (UID: "09456d5f-e8fe-432a-b11f-093f94a8a8e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:47:18.901213 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.901197 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:47:18.903140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.903110 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca" pod "09456d5f-e8fe-432a-b11f-093f94a8a8e7" (UID: "09456d5f-e8fe-432a-b11f-093f94a8a8e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:47:18.903248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:18.903174 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6" pod "09456d5f-e8fe-432a-b11f-093f94a8a8e7" (UID: "09456d5f-e8fe-432a-b11f-093f94a8a8e7"). InnerVolumeSpecName "kube-api-access-44vt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:47:19.001776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:19.001696 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09456d5f-e8fe-432a-b11f-093f94a8a8e7-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:47:19.001776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:19.001722 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44vt6\" (UniqueName: \"kubernetes.io/projected/09456d5f-e8fe-432a-b11f-093f94a8a8e7-kube-api-access-44vt6\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:47:19.678879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:19.678850 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" Nov 03 01:47:19.678879 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:19.678863 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4" event={"ID":"09456d5f-e8fe-432a-b11f-093f94a8a8e7","Type":"ContainerDied","Data":"9bada44aca188256d9490b975aec9ad03222deabe6bb4ec591d8612580023fae"} Nov 03 01:47:19.679104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:19.678908 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bada44aca188256d9490b975aec9ad03222deabe6bb4ec591d8612580023fae" Nov 03 01:47:42.162485 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:47:42.162424 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:47:48.048001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:48.047965 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d"] Nov 03 01:47:48.097529 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:48.097483 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b4n9d"] Nov 03 01:47:49.997676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:47:49.997543 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce5ac2f-9c1e-48da-b458-54e76b8ce391" path="/var/lib/kubelet/pods/5ce5ac2f-9c1e-48da-b458-54e76b8ce391/volumes" Nov 03 01:48:03.725687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.725655 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6"] Nov 03 01:48:03.726138 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.726125 5144 cpu_manager.go:414] "containerMap: removing container" podUID="09456d5f-e8fe-432a-b11f-093f94a8a8e7" containerName="revoke-keys" Nov 03 01:48:03.726185 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.726139 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09456d5f-e8fe-432a-b11f-093f94a8a8e7" containerName="revoke-keys" Nov 03 01:48:03.726228 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.726215 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="09456d5f-e8fe-432a-b11f-093f94a8a8e7" containerName="revoke-keys" Nov 03 01:48:03.729481 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.729463 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.785996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.785962 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6"] Nov 03 01:48:03.831722 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.831686 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:48:03.831921 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.831772 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:48:03.840243 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.840201 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpt8d\" (UniqueName: \"kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.840416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.840379 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.840416 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.840408 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.941381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.941337 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.941381 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.941379 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.941626 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.941419 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpt8d\" (UniqueName: \"kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.942072 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.942043 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.944001 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.943980 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:03.998423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:03.998339 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpt8d\" (UniqueName: \"kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d\") pod \"maas-api-revoke-sub-d3b8d90c-qkbb6\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:04.042406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:04.042362 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:04.419930 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:04.419898 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6"] Nov 03 01:48:04.421291 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:48:04.421252 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e2bb2fd_c4fc_4e36_9a6b_5b4242d1382c.slice/crio-614294c0938dd13c29da6726dac7e4d748297f8d517851c1ecad567ca5da9f31 WatchSource:0}: Error finding container 614294c0938dd13c29da6726dac7e4d748297f8d517851c1ecad567ca5da9f31: Status 404 returned error can't find the container with id 614294c0938dd13c29da6726dac7e4d748297f8d517851c1ecad567ca5da9f31 Nov 03 01:48:05.050149 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:05.050107 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" event={"ID":"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c","Type":"ContainerStarted","Data":"c780533db740804a5d45522c231b091824764e6ce8aaba557bef9f94303098db"} Nov 03 01:48:05.050779 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:05.050165 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" event={"ID":"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c","Type":"ContainerStarted","Data":"614294c0938dd13c29da6726dac7e4d748297f8d517851c1ecad567ca5da9f31"} Nov 03 01:48:05.165871 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:05.165807 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" podStartSLOduration=2.165792023 podStartE2EDuration="2.165792023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:05.162733677 +0000 UTC m=+6175.308428140" watchObservedRunningTime="2026-11-03 01:48:05.165792023 +0000 UTC m=+6175.311486484" Nov 03 01:48:06.058277 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:06.058239 5144 generic.go:356] "Generic (PLEG): container finished" podID="2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" containerID="c780533db740804a5d45522c231b091824764e6ce8aaba557bef9f94303098db" exitCode=0 Nov 03 01:48:06.058818 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:06.058302 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" event={"ID":"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c","Type":"ContainerDied","Data":"c780533db740804a5d45522c231b091824764e6ce8aaba557bef9f94303098db"} Nov 03 01:48:07.219366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.219337 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:07.372857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.372755 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca\") pod \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " Nov 03 01:48:07.372857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.372823 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d\" (UniqueName: \"kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d\") pod \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " Nov 03 01:48:07.373130 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.372916 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca\") pod \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\" (UID: \"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c\") " Nov 03 01:48:07.373203 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.373160 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca" pod "2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" (UID: "2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:07.375432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.375392 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca" pod "2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" (UID: "2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:07.375432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.375412 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d" pod "2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" (UID: "2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c"). InnerVolumeSpecName "kube-api-access-tpt8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:07.474186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.474128 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:48:07.474406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.474195 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:48:07.474406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:07.474214 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpt8d\" (UniqueName: \"kubernetes.io/projected/2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c-kube-api-access-tpt8d\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:48:08.072199 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:08.072166 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" Nov 03 01:48:08.072398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:08.072206 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3b8d90c-qkbb6" event={"ID":"2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c","Type":"ContainerDied","Data":"614294c0938dd13c29da6726dac7e4d748297f8d517851c1ecad567ca5da9f31"} Nov 03 01:48:08.072398 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:08.072255 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="614294c0938dd13c29da6726dac7e4d748297f8d517851c1ecad567ca5da9f31" Nov 03 01:48:12.157702 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:48:12.157666 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:48:14.486084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.486040 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp"] Nov 03 01:48:14.486899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.486875 5144 cpu_manager.go:414] "containerMap: removing container" podUID="2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" containerName="revoke-keys" Nov 03 01:48:14.486969 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.486902 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" containerName="revoke-keys" Nov 03 01:48:14.487065 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.487054 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c" containerName="revoke-keys" Nov 03 01:48:14.491087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.491059 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.545087 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.545046 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp"] Nov 03 01:48:14.590765 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.590732 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:48:14.590955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.590825 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:48:14.650776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.650739 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.650776 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.650780 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66tz\" (UniqueName: \"kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.651030 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.650884 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.752186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.752089 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.752383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.752238 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.752383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.752271 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66tz\" (UniqueName: \"kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.752830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.752806 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.754866 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.754836 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:14.809738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:14.809700 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66tz\" (UniqueName: \"kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz\") pod \"maas-api-revoke-sub-b3655629-4n7hp\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:15.105613 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:15.105576 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:15.195485 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:15.195456 5144 scope.go:122] "RemoveContainer" containerID="b176ee92ac2aab47a4c99d12340c71ed7b2350c7a6d4369b22880847e1c35486" Nov 03 01:48:15.484696 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:15.484669 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp"] Nov 03 01:48:15.486156 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:48:15.486125 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a715a7_85d4_4567_b549_12daf9b79983.slice/crio-82d8e2d295735a7517553e29c7e3f2a0a4a662135ac761d7be394bdf1779808a WatchSource:0}: Error finding container 82d8e2d295735a7517553e29c7e3f2a0a4a662135ac761d7be394bdf1779808a: Status 404 returned error can't find the container with id 82d8e2d295735a7517553e29c7e3f2a0a4a662135ac761d7be394bdf1779808a Nov 03 01:48:16.134121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:16.134015 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" event={"ID":"58a715a7-85d4-4567-b549-12daf9b79983","Type":"ContainerStarted","Data":"fd8ae83e5ced0e165c955be2ed831dab9edfe12981eda09dcc2e7caafcb5170b"} Nov 03 01:48:16.134121 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:16.134078 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" event={"ID":"58a715a7-85d4-4567-b549-12daf9b79983","Type":"ContainerStarted","Data":"82d8e2d295735a7517553e29c7e3f2a0a4a662135ac761d7be394bdf1779808a"} Nov 03 01:48:16.248312 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:16.248256 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" podStartSLOduration=2.248238453 podStartE2EDuration="2.248238453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:48:16.245553314 +0000 UTC m=+6186.391247784" watchObservedRunningTime="2026-11-03 01:48:16.248238453 +0000 UTC m=+6186.393932910" Nov 03 01:48:17.144118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:17.144076 5144 generic.go:356] "Generic (PLEG): container finished" podID="58a715a7-85d4-4567-b549-12daf9b79983" containerID="fd8ae83e5ced0e165c955be2ed831dab9edfe12981eda09dcc2e7caafcb5170b" exitCode=0 Nov 03 01:48:17.144732 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:17.144171 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" event={"ID":"58a715a7-85d4-4567-b549-12daf9b79983","Type":"ContainerDied","Data":"fd8ae83e5ced0e165c955be2ed831dab9edfe12981eda09dcc2e7caafcb5170b"} Nov 03 01:48:18.589136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.589098 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:18.602891 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.602848 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca\") pod \"58a715a7-85d4-4567-b549-12daf9b79983\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " Nov 03 01:48:18.603111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.602970 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz\" (UniqueName: \"kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz\") pod \"58a715a7-85d4-4567-b549-12daf9b79983\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " Nov 03 01:48:18.603111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.603035 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca\") pod \"58a715a7-85d4-4567-b549-12daf9b79983\" (UID: \"58a715a7-85d4-4567-b549-12daf9b79983\") " Nov 03 01:48:18.603432 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.603400 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca" pod "58a715a7-85d4-4567-b549-12daf9b79983" (UID: "58a715a7-85d4-4567-b549-12daf9b79983"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:48:18.605190 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.605159 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca" pod "58a715a7-85d4-4567-b549-12daf9b79983" (UID: "58a715a7-85d4-4567-b549-12daf9b79983"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:48:18.605304 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.605223 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz" pod "58a715a7-85d4-4567-b549-12daf9b79983" (UID: "58a715a7-85d4-4567-b549-12daf9b79983"). InnerVolumeSpecName "kube-api-access-z66tz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:48:18.704022 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.703980 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66tz\" (UniqueName: \"kubernetes.io/projected/58a715a7-85d4-4567-b549-12daf9b79983-kube-api-access-z66tz\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:48:18.704022 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.704012 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:48:18.704022 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:18.704026 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/58a715a7-85d4-4567-b549-12daf9b79983-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:48:19.197324 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:19.197276 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" event={"ID":"58a715a7-85d4-4567-b549-12daf9b79983","Type":"ContainerDied","Data":"82d8e2d295735a7517553e29c7e3f2a0a4a662135ac761d7be394bdf1779808a"} Nov 03 01:48:19.197653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:19.197341 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82d8e2d295735a7517553e29c7e3f2a0a4a662135ac761d7be394bdf1779808a" Nov 03 01:48:19.197653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:48:19.197498 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3655629-4n7hp" Nov 03 01:48:42.156339 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:48:42.156295 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:49:05.634670 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.634592 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r"] Nov 03 01:49:05.635273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.635149 5144 cpu_manager.go:414] "containerMap: removing container" podUID="58a715a7-85d4-4567-b549-12daf9b79983" containerName="revoke-keys" Nov 03 01:49:05.635273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.635162 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a715a7-85d4-4567-b549-12daf9b79983" containerName="revoke-keys" Nov 03 01:49:05.635273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.635265 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a715a7-85d4-4567-b549-12daf9b79983" containerName="revoke-keys" Nov 03 01:49:05.638246 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.638225 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.694452 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.694412 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r"] Nov 03 01:49:05.740376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.740337 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:49:05.740376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.740371 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:49:05.762329 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.762279 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmjkp\" (UniqueName: \"kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.762567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.762345 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.762567 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.762466 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.863898 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.863862 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmjkp\" (UniqueName: \"kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.863898 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.863908 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.864210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.863945 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.864562 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.864541 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.866904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.866878 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.920369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.920290 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmjkp\" (UniqueName: \"kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp\") pod \"maas-api-revoke-sub-dbe653a3-72l7r\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:05.951778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:05.951744 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:06.327152 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:06.327120 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r"] Nov 03 01:49:06.328337 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:49:06.328307 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod720fc652_4219_49c4_9156_ce5ddff6d188.slice/crio-d06d68078b82338fae795798657edf1bf7c06eb89ed67c898cc8c7bc0f9cff6d WatchSource:0}: Error finding container d06d68078b82338fae795798657edf1bf7c06eb89ed67c898cc8c7bc0f9cff6d: Status 404 returned error can't find the container with id d06d68078b82338fae795798657edf1bf7c06eb89ed67c898cc8c7bc0f9cff6d Nov 03 01:49:06.560908 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:06.560864 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" event={"ID":"720fc652-4219-49c4-9156-ce5ddff6d188","Type":"ContainerStarted","Data":"d06d68078b82338fae795798657edf1bf7c06eb89ed67c898cc8c7bc0f9cff6d"} Nov 03 01:49:07.570686 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:07.570647 5144 generic.go:356] "Generic (PLEG): container finished" podID="720fc652-4219-49c4-9156-ce5ddff6d188" containerID="91db69b422b9ed94db794bb55af78513a3348522a15188f1f18e192f0b611adf" exitCode=0 Nov 03 01:49:07.571163 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:07.570730 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" event={"ID":"720fc652-4219-49c4-9156-ce5ddff6d188","Type":"ContainerDied","Data":"91db69b422b9ed94db794bb55af78513a3348522a15188f1f18e192f0b611adf"} Nov 03 01:49:08.729892 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.729860 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:08.794580 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.794547 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca\") pod \"720fc652-4219-49c4-9156-ce5ddff6d188\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " Nov 03 01:49:08.794832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.794607 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca\") pod \"720fc652-4219-49c4-9156-ce5ddff6d188\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " Nov 03 01:49:08.794832 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.794709 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp\" (UniqueName: \"kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp\") pod \"720fc652-4219-49c4-9156-ce5ddff6d188\" (UID: \"720fc652-4219-49c4-9156-ce5ddff6d188\") " Nov 03 01:49:08.795014 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.794991 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca" pod "720fc652-4219-49c4-9156-ce5ddff6d188" (UID: "720fc652-4219-49c4-9156-ce5ddff6d188"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:08.795095 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.795079 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:08.797086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.797057 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca" pod "720fc652-4219-49c4-9156-ce5ddff6d188" (UID: "720fc652-4219-49c4-9156-ce5ddff6d188"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:08.797086 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.797069 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp" pod "720fc652-4219-49c4-9156-ce5ddff6d188" (UID: "720fc652-4219-49c4-9156-ce5ddff6d188"). InnerVolumeSpecName "kube-api-access-wmjkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:08.895812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.895705 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmjkp\" (UniqueName: \"kubernetes.io/projected/720fc652-4219-49c4-9156-ce5ddff6d188-kube-api-access-wmjkp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:08.895812 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:08.895745 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/720fc652-4219-49c4-9156-ce5ddff6d188-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:09.585936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:09.585901 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" Nov 03 01:49:09.585936 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:09.585931 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dbe653a3-72l7r" event={"ID":"720fc652-4219-49c4-9156-ce5ddff6d188","Type":"ContainerDied","Data":"d06d68078b82338fae795798657edf1bf7c06eb89ed67c898cc8c7bc0f9cff6d"} Nov 03 01:49:09.586194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:09.585989 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d06d68078b82338fae795798657edf1bf7c06eb89ed67c898cc8c7bc0f9cff6d" Nov 03 01:49:12.158126 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:49:12.158087 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:49:15.223102 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:15.223070 5144 scope.go:122] "RemoveContainer" containerID="50a8ab4fd034e868a48c4c05223c1f58986d34fe7db1bdf8ccb26364e78713ea" Nov 03 01:49:16.823676 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.823620 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v"] Nov 03 01:49:16.824227 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.824169 5144 cpu_manager.go:414] "containerMap: removing container" podUID="720fc652-4219-49c4-9156-ce5ddff6d188" containerName="revoke-keys" Nov 03 01:49:16.824227 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.824182 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="720fc652-4219-49c4-9156-ce5ddff6d188" containerName="revoke-keys" Nov 03 01:49:16.824355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.824262 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="720fc652-4219-49c4-9156-ce5ddff6d188" containerName="revoke-keys" Nov 03 01:49:16.829098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.829069 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:16.883942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.883905 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v"] Nov 03 01:49:16.919451 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.919414 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:49:16.929519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.929490 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:49:16.980748 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.980706 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:16.980963 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.980775 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5pjt\" (UniqueName: \"kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:16.980963 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:16.980922 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.082170 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.082068 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.082170 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.082164 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5pjt\" (UniqueName: \"kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.082462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.082239 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.082958 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.082933 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.084813 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.084789 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.139422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.139382 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5pjt\" (UniqueName: \"kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt\") pod \"maas-api-revoke-sub-f341e88a-x554v\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.144252 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.144219 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:17.531339 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.531305 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v"] Nov 03 01:49:17.532855 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:49:17.532824 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f8ca20_c82c_414f_9ea8_658d3c692053.slice/crio-309e1e1632c1f0ffacec87ccb81f0b26b29a04417381948972fc52feac6bac51 WatchSource:0}: Error finding container 309e1e1632c1f0ffacec87ccb81f0b26b29a04417381948972fc52feac6bac51: Status 404 returned error can't find the container with id 309e1e1632c1f0ffacec87ccb81f0b26b29a04417381948972fc52feac6bac51 Nov 03 01:49:17.637932 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:17.637896 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" event={"ID":"00f8ca20-c82c-414f-9ea8-658d3c692053","Type":"ContainerStarted","Data":"309e1e1632c1f0ffacec87ccb81f0b26b29a04417381948972fc52feac6bac51"} Nov 03 01:49:18.646261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:18.646221 5144 generic.go:356] "Generic (PLEG): container finished" podID="00f8ca20-c82c-414f-9ea8-658d3c692053" containerID="f4308b1a62ed54701dba1f3c8b84700cdda3da8896af31bd3b9181c7329c0453" exitCode=0 Nov 03 01:49:18.646740 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:18.646318 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" event={"ID":"00f8ca20-c82c-414f-9ea8-658d3c692053","Type":"ContainerDied","Data":"f4308b1a62ed54701dba1f3c8b84700cdda3da8896af31bd3b9181c7329c0453"} Nov 03 01:49:19.832461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.832427 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:19.913422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.913318 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt\" (UniqueName: \"kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt\") pod \"00f8ca20-c82c-414f-9ea8-658d3c692053\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " Nov 03 01:49:19.913674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.913429 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca\") pod \"00f8ca20-c82c-414f-9ea8-658d3c692053\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " Nov 03 01:49:19.913674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.913478 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca\") pod \"00f8ca20-c82c-414f-9ea8-658d3c692053\" (UID: \"00f8ca20-c82c-414f-9ea8-658d3c692053\") " Nov 03 01:49:19.913899 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.913867 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca" pod "00f8ca20-c82c-414f-9ea8-658d3c692053" (UID: "00f8ca20-c82c-414f-9ea8-658d3c692053"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:19.915980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.915943 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca" pod "00f8ca20-c82c-414f-9ea8-658d3c692053" (UID: "00f8ca20-c82c-414f-9ea8-658d3c692053"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:19.916100 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:19.916037 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt" pod "00f8ca20-c82c-414f-9ea8-658d3c692053" (UID: "00f8ca20-c82c-414f-9ea8-658d3c692053"). InnerVolumeSpecName "kube-api-access-f5pjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:20.015116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:20.015078 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:20.015116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:20.015115 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00f8ca20-c82c-414f-9ea8-658d3c692053-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:20.015116 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:20.015125 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5pjt\" (UniqueName: \"kubernetes.io/projected/00f8ca20-c82c-414f-9ea8-658d3c692053-kube-api-access-f5pjt\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:20.660273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:20.660234 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" Nov 03 01:49:20.660273 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:20.660267 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f341e88a-x554v" event={"ID":"00f8ca20-c82c-414f-9ea8-658d3c692053","Type":"ContainerDied","Data":"309e1e1632c1f0ffacec87ccb81f0b26b29a04417381948972fc52feac6bac51"} Nov 03 01:49:20.660579 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:20.660352 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="309e1e1632c1f0ffacec87ccb81f0b26b29a04417381948972fc52feac6bac51" Nov 03 01:49:27.842944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.842896 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p"] Nov 03 01:49:27.843602 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.843579 5144 cpu_manager.go:414] "containerMap: removing container" podUID="00f8ca20-c82c-414f-9ea8-658d3c692053" containerName="revoke-keys" Nov 03 01:49:27.843757 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.843604 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f8ca20-c82c-414f-9ea8-658d3c692053" containerName="revoke-keys" Nov 03 01:49:27.843825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.843759 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f8ca20-c82c-414f-9ea8-658d3c692053" containerName="revoke-keys" Nov 03 01:49:27.847342 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.847317 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:27.902115 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.902076 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p"] Nov 03 01:49:27.947886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.947845 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:49:27.947886 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:27.947854 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:49:28.035435 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.035391 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.035681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.035520 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzp25\" (UniqueName: \"kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.035681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.035623 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.137188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.137084 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzp25\" (UniqueName: \"kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.137372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.137189 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.137372 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.137232 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.137982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.137954 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.139928 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.139899 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.194917 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.194863 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzp25\" (UniqueName: \"kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25\") pod \"maas-api-revoke-sub-d2614c30-f7d6p\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.461878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.461778 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:28.844274 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:28.844235 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p"] Nov 03 01:49:28.845421 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:49:28.845388 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb362b89_1a32_4f21_a6a8_52af65337218.slice/crio-d579d0a08e208041fea7be792f4fd8a401a25316bbcd07938f869440b0218fc1 WatchSource:0}: Error finding container d579d0a08e208041fea7be792f4fd8a401a25316bbcd07938f869440b0218fc1: Status 404 returned error can't find the container with id d579d0a08e208041fea7be792f4fd8a401a25316bbcd07938f869440b0218fc1 Nov 03 01:49:29.741314 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:29.741276 5144 generic.go:356] "Generic (PLEG): container finished" podID="eb362b89-1a32-4f21-a6a8-52af65337218" containerID="bfcf4b1715aa3aa08b7b2b395107c6ba1fc0bafb44a50f7fedc22ff1d021fc34" exitCode=0 Nov 03 01:49:29.741540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:29.741355 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" event={"ID":"eb362b89-1a32-4f21-a6a8-52af65337218","Type":"ContainerDied","Data":"bfcf4b1715aa3aa08b7b2b395107c6ba1fc0bafb44a50f7fedc22ff1d021fc34"} Nov 03 01:49:29.741540 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:29.741413 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" event={"ID":"eb362b89-1a32-4f21-a6a8-52af65337218","Type":"ContainerStarted","Data":"d579d0a08e208041fea7be792f4fd8a401a25316bbcd07938f869440b0218fc1"} Nov 03 01:49:30.894481 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:30.894450 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:31.069438 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.069397 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca\") pod \"eb362b89-1a32-4f21-a6a8-52af65337218\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " Nov 03 01:49:31.069707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.069533 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25\" (UniqueName: \"kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25\") pod \"eb362b89-1a32-4f21-a6a8-52af65337218\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " Nov 03 01:49:31.069707 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.069573 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca\") pod \"eb362b89-1a32-4f21-a6a8-52af65337218\" (UID: \"eb362b89-1a32-4f21-a6a8-52af65337218\") " Nov 03 01:49:31.069938 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.069912 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca" pod "eb362b89-1a32-4f21-a6a8-52af65337218" (UID: "eb362b89-1a32-4f21-a6a8-52af65337218"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:31.071996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.071970 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca" pod "eb362b89-1a32-4f21-a6a8-52af65337218" (UID: "eb362b89-1a32-4f21-a6a8-52af65337218"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:31.072177 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.071997 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25" pod "eb362b89-1a32-4f21-a6a8-52af65337218" (UID: "eb362b89-1a32-4f21-a6a8-52af65337218"). InnerVolumeSpecName "kube-api-access-wzp25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:31.170770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.170723 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzp25\" (UniqueName: \"kubernetes.io/projected/eb362b89-1a32-4f21-a6a8-52af65337218-kube-api-access-wzp25\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:31.170770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.170770 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:31.171008 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.170787 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb362b89-1a32-4f21-a6a8-52af65337218-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:31.764191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.764150 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" Nov 03 01:49:31.764191 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.764170 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2614c30-f7d6p" event={"ID":"eb362b89-1a32-4f21-a6a8-52af65337218","Type":"ContainerDied","Data":"d579d0a08e208041fea7be792f4fd8a401a25316bbcd07938f869440b0218fc1"} Nov 03 01:49:31.764604 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:31.764254 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d579d0a08e208041fea7be792f4fd8a401a25316bbcd07938f869440b0218fc1" Nov 03 01:49:38.774802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.774766 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz"] Nov 03 01:49:38.775364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.775288 5144 cpu_manager.go:414] "containerMap: removing container" podUID="eb362b89-1a32-4f21-a6a8-52af65337218" containerName="revoke-keys" Nov 03 01:49:38.775364 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.775302 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb362b89-1a32-4f21-a6a8-52af65337218" containerName="revoke-keys" Nov 03 01:49:38.775525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.775382 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb362b89-1a32-4f21-a6a8-52af65337218" containerName="revoke-keys" Nov 03 01:49:38.778933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.778903 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.833391 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.833356 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz"] Nov 03 01:49:38.845250 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.845213 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2htrx\" (UniqueName: \"kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.845467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.845259 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.845467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.845420 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.879060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.879025 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:49:38.879060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.879041 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:49:38.946378 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.946316 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.946619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.946428 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2htrx\" (UniqueName: \"kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.946619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.946455 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.947210 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.947186 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:38.949119 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:38.949088 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:39.003279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.003246 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2htrx\" (UniqueName: \"kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx\") pod \"maas-api-revoke-sub-ced98d90-2mspz\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:39.044358 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.044260 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2"] Nov 03 01:49:39.048305 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.048267 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-5vjz2"] Nov 03 01:49:39.096981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.096943 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:39.474039 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.474012 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz"] Nov 03 01:49:39.475572 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:49:39.475545 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod417201ed_aeba_46ec_acb2_281076d53f6f.slice/crio-ea4e2476970b92264cbf8f43301448c394c0d70b89af5d6358af7efb844358ac WatchSource:0}: Error finding container ea4e2476970b92264cbf8f43301448c394c0d70b89af5d6358af7efb844358ac: Status 404 returned error can't find the container with id ea4e2476970b92264cbf8f43301448c394c0d70b89af5d6358af7efb844358ac Nov 03 01:49:39.820464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.820407 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" event={"ID":"417201ed-aeba-46ec-acb2-281076d53f6f","Type":"ContainerStarted","Data":"23ee2ea60ac0c53ba9aeb2cf25d4320978b2fb5223133070ffa454ba679db3c2"} Nov 03 01:49:39.820979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.820478 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" event={"ID":"417201ed-aeba-46ec-acb2-281076d53f6f","Type":"ContainerStarted","Data":"ea4e2476970b92264cbf8f43301448c394c0d70b89af5d6358af7efb844358ac"} Nov 03 01:49:39.937324 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.937263 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" podStartSLOduration=1.937247055 podStartE2EDuration="1.937247055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:39.934163329 +0000 UTC m=+6270.079857810" watchObservedRunningTime="2026-11-03 01:49:39.937247055 +0000 UTC m=+6270.082941511" Nov 03 01:49:39.996944 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:39.996854 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9f1fca8-2189-4aed-81a6-d16512f882de" path="/var/lib/kubelet/pods/a9f1fca8-2189-4aed-81a6-d16512f882de/volumes" Nov 03 01:49:40.828344 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:40.828309 5144 generic.go:356] "Generic (PLEG): container finished" podID="417201ed-aeba-46ec-acb2-281076d53f6f" containerID="23ee2ea60ac0c53ba9aeb2cf25d4320978b2fb5223133070ffa454ba679db3c2" exitCode=0 Nov 03 01:49:40.828822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:40.828368 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" event={"ID":"417201ed-aeba-46ec-acb2-281076d53f6f","Type":"ContainerDied","Data":"23ee2ea60ac0c53ba9aeb2cf25d4320978b2fb5223133070ffa454ba679db3c2"} Nov 03 01:49:42.001006 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.000977 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:42.079677 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.079607 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca\") pod \"417201ed-aeba-46ec-acb2-281076d53f6f\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " Nov 03 01:49:42.079881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.079757 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx\" (UniqueName: \"kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx\") pod \"417201ed-aeba-46ec-acb2-281076d53f6f\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " Nov 03 01:49:42.079881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.079811 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca\") pod \"417201ed-aeba-46ec-acb2-281076d53f6f\" (UID: \"417201ed-aeba-46ec-acb2-281076d53f6f\") " Nov 03 01:49:42.079988 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.079967 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca" pod "417201ed-aeba-46ec-acb2-281076d53f6f" (UID: "417201ed-aeba-46ec-acb2-281076d53f6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:42.080217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.080198 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:42.082204 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.082180 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca" pod "417201ed-aeba-46ec-acb2-281076d53f6f" (UID: "417201ed-aeba-46ec-acb2-281076d53f6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:42.082349 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.082219 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx" pod "417201ed-aeba-46ec-acb2-281076d53f6f" (UID: "417201ed-aeba-46ec-acb2-281076d53f6f"). InnerVolumeSpecName "kube-api-access-2htrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:42.161015 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:49:42.160950 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:49:42.181125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.181086 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/417201ed-aeba-46ec-acb2-281076d53f6f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:42.181125 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.181121 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2htrx\" (UniqueName: \"kubernetes.io/projected/417201ed-aeba-46ec-acb2-281076d53f6f-kube-api-access-2htrx\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:42.842094 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.842056 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" Nov 03 01:49:42.842310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.842093 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ced98d90-2mspz" event={"ID":"417201ed-aeba-46ec-acb2-281076d53f6f","Type":"ContainerDied","Data":"ea4e2476970b92264cbf8f43301448c394c0d70b89af5d6358af7efb844358ac"} Nov 03 01:49:42.842310 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:42.842182 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea4e2476970b92264cbf8f43301448c394c0d70b89af5d6358af7efb844358ac" Nov 03 01:49:49.485864 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.485824 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz"] Nov 03 01:49:49.486348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.486322 5144 cpu_manager.go:414] "containerMap: removing container" podUID="417201ed-aeba-46ec-acb2-281076d53f6f" containerName="revoke-keys" Nov 03 01:49:49.486348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.486334 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="417201ed-aeba-46ec-acb2-281076d53f6f" containerName="revoke-keys" Nov 03 01:49:49.486446 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.486416 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="417201ed-aeba-46ec-acb2-281076d53f6f" containerName="revoke-keys" Nov 03 01:49:49.489996 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.489968 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.545226 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.545188 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz"] Nov 03 01:49:49.561033 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.560991 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.561251 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.561140 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.561315 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.561269 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjnck\" (UniqueName: \"kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.590858 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.590814 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:49:49.591055 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.590860 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:49:49.663038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.662998 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.663252 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.663058 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjnck\" (UniqueName: \"kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.663252 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.663151 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.663948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.663867 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.665789 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.665766 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.719829 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.719801 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjnck\" (UniqueName: \"kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck\") pod \"maas-api-revoke-sub-a211ca3a-vw4fz\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:49.809844 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:49.809797 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:50.186916 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:50.186889 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz"] Nov 03 01:49:50.188583 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:49:50.188547 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1331ac_1173_44a5_b7e5_06f314e938b1.slice/crio-4063c5fd3f5c443ddbce98328aa978135ad3f80852f514f1d63b0733bf93345e WatchSource:0}: Error finding container 4063c5fd3f5c443ddbce98328aa978135ad3f80852f514f1d63b0733bf93345e: Status 404 returned error can't find the container with id 4063c5fd3f5c443ddbce98328aa978135ad3f80852f514f1d63b0733bf93345e Nov 03 01:49:50.920976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:50.920873 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" event={"ID":"3d1331ac-1173-44a5-b7e5-06f314e938b1","Type":"ContainerStarted","Data":"aceced016d642b887fe70c8391ff78ef283e8a17f4f6592a2c723b3f86811630"} Nov 03 01:49:50.920976 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:50.920945 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" event={"ID":"3d1331ac-1173-44a5-b7e5-06f314e938b1","Type":"ContainerStarted","Data":"4063c5fd3f5c443ddbce98328aa978135ad3f80852f514f1d63b0733bf93345e"} Nov 03 01:49:51.035284 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:51.035224 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" podStartSLOduration=2.035205616 podStartE2EDuration="2.035205616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:49:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:49:51.033152905 +0000 UTC m=+6281.178847387" watchObservedRunningTime="2026-11-03 01:49:51.035205616 +0000 UTC m=+6281.180900051" Nov 03 01:49:51.930076 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:51.930042 5144 generic.go:356] "Generic (PLEG): container finished" podID="3d1331ac-1173-44a5-b7e5-06f314e938b1" containerID="aceced016d642b887fe70c8391ff78ef283e8a17f4f6592a2c723b3f86811630" exitCode=0 Nov 03 01:49:51.930619 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:51.930122 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" event={"ID":"3d1331ac-1173-44a5-b7e5-06f314e938b1","Type":"ContainerDied","Data":"aceced016d642b887fe70c8391ff78ef283e8a17f4f6592a2c723b3f86811630"} Nov 03 01:49:53.091732 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.091702 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:53.201693 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.201559 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca\") pod \"3d1331ac-1173-44a5-b7e5-06f314e938b1\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " Nov 03 01:49:53.201881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.201715 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck\" (UniqueName: \"kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck\") pod \"3d1331ac-1173-44a5-b7e5-06f314e938b1\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " Nov 03 01:49:53.201881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.201793 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca\") pod \"3d1331ac-1173-44a5-b7e5-06f314e938b1\" (UID: \"3d1331ac-1173-44a5-b7e5-06f314e938b1\") " Nov 03 01:49:53.202085 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.202059 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca" pod "3d1331ac-1173-44a5-b7e5-06f314e938b1" (UID: "3d1331ac-1173-44a5-b7e5-06f314e938b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:49:53.204588 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.204552 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck" pod "3d1331ac-1173-44a5-b7e5-06f314e938b1" (UID: "3d1331ac-1173-44a5-b7e5-06f314e938b1"). InnerVolumeSpecName "kube-api-access-bjnck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:49:53.206781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.206751 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca" pod "3d1331ac-1173-44a5-b7e5-06f314e938b1" (UID: "3d1331ac-1173-44a5-b7e5-06f314e938b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:49:53.303467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.303409 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:53.303467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.303461 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1331ac-1173-44a5-b7e5-06f314e938b1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:53.303467 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.303471 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjnck\" (UniqueName: \"kubernetes.io/projected/3d1331ac-1173-44a5-b7e5-06f314e938b1-kube-api-access-bjnck\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:49:53.943678 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.943645 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" Nov 03 01:49:53.943881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.943672 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a211ca3a-vw4fz" event={"ID":"3d1331ac-1173-44a5-b7e5-06f314e938b1","Type":"ContainerDied","Data":"4063c5fd3f5c443ddbce98328aa978135ad3f80852f514f1d63b0733bf93345e"} Nov 03 01:49:53.943881 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:49:53.943748 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4063c5fd3f5c443ddbce98328aa978135ad3f80852f514f1d63b0733bf93345e" Nov 03 01:50:00.051424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.051387 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr"] Nov 03 01:50:00.101258 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.101217 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-nqrhr"] Nov 03 01:50:00.460563 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.460469 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc"] Nov 03 01:50:00.460975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.460961 5144 cpu_manager.go:414] "containerMap: removing container" podUID="3d1331ac-1173-44a5-b7e5-06f314e938b1" containerName="revoke-keys" Nov 03 01:50:00.461024 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.460977 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1331ac-1173-44a5-b7e5-06f314e938b1" containerName="revoke-keys" Nov 03 01:50:00.461070 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.461061 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1331ac-1173-44a5-b7e5-06f314e938b1" containerName="revoke-keys" Nov 03 01:50:00.464433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.464414 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.519599 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.519563 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc"] Nov 03 01:50:00.565820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.565778 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:00.565820 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.565801 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:50:00.582380 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.582335 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl2b5\" (UniqueName: \"kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.582571 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.582459 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.582571 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.582497 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.683521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.683482 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl2b5\" (UniqueName: \"kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.683738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.683563 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.683738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.683583 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.684319 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.684286 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.686519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.686483 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.740555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.740476 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl2b5\" (UniqueName: \"kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5\") pod \"maas-api-revoke-sub-69306e62-dc8fc\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:00.778186 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:00.778144 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:01.158035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:01.158002 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc"] Nov 03 01:50:01.159539 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:50:01.159513 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f83fe52_aeb6_4b53_82aa_b3734985eb2f.slice/crio-e8532fe271f091974ba738c7f5661cbb21866c068b2527826fcef9b3ca3fd0d1 WatchSource:0}: Error finding container e8532fe271f091974ba738c7f5661cbb21866c068b2527826fcef9b3ca3fd0d1: Status 404 returned error can't find the container with id e8532fe271f091974ba738c7f5661cbb21866c068b2527826fcef9b3ca3fd0d1 Nov 03 01:50:01.995862 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:01.995826 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f426b483-f5c1-4dc7-afcd-7118d77468fc" path="/var/lib/kubelet/pods/f426b483-f5c1-4dc7-afcd-7118d77468fc/volumes" Nov 03 01:50:01.996151 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:01.996133 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" event={"ID":"0f83fe52-aeb6-4b53-82aa-b3734985eb2f","Type":"ContainerStarted","Data":"a539f21fbf786bd45c155f61cca1fd4a32040f2144f9a416bfd836f05c5289f7"} Nov 03 01:50:01.996207 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:01.996194 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" event={"ID":"0f83fe52-aeb6-4b53-82aa-b3734985eb2f","Type":"ContainerStarted","Data":"e8532fe271f091974ba738c7f5661cbb21866c068b2527826fcef9b3ca3fd0d1"} Nov 03 01:50:02.110408 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:02.110324 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" podStartSLOduration=2.1103008 podStartE2EDuration="2.1103008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:02.106348085 +0000 UTC m=+6292.252042565" watchObservedRunningTime="2026-11-03 01:50:02.1103008 +0000 UTC m=+6292.255995259" Nov 03 01:50:03.004905 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:03.004861 5144 generic.go:356] "Generic (PLEG): container finished" podID="0f83fe52-aeb6-4b53-82aa-b3734985eb2f" containerID="a539f21fbf786bd45c155f61cca1fd4a32040f2144f9a416bfd836f05c5289f7" exitCode=0 Nov 03 01:50:03.005623 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:03.004959 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" event={"ID":"0f83fe52-aeb6-4b53-82aa-b3734985eb2f","Type":"ContainerDied","Data":"a539f21fbf786bd45c155f61cca1fd4a32040f2144f9a416bfd836f05c5289f7"} Nov 03 01:50:04.434528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.434107 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:04.495118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.495069 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca\") pod \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " Nov 03 01:50:04.495371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.495127 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5\" (UniqueName: \"kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5\") pod \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " Nov 03 01:50:04.495371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.495197 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca\") pod \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\" (UID: \"0f83fe52-aeb6-4b53-82aa-b3734985eb2f\") " Nov 03 01:50:04.497909 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.497867 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca" pod "0f83fe52-aeb6-4b53-82aa-b3734985eb2f" (UID: "0f83fe52-aeb6-4b53-82aa-b3734985eb2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:04.500837 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.500796 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5" pod "0f83fe52-aeb6-4b53-82aa-b3734985eb2f" (UID: "0f83fe52-aeb6-4b53-82aa-b3734985eb2f"). InnerVolumeSpecName "kube-api-access-bl2b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:04.501584 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.501548 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca" pod "0f83fe52-aeb6-4b53-82aa-b3734985eb2f" (UID: "0f83fe52-aeb6-4b53-82aa-b3734985eb2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:04.603968 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.603919 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:04.604276 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.604253 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:04.604433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:04.604420 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl2b5\" (UniqueName: \"kubernetes.io/projected/0f83fe52-aeb6-4b53-82aa-b3734985eb2f-kube-api-access-bl2b5\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:05.036675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:05.035752 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:50:05.036675 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:05.035833 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:50:05.048477 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:05.048435 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" event={"ID":"0f83fe52-aeb6-4b53-82aa-b3734985eb2f","Type":"ContainerDied","Data":"e8532fe271f091974ba738c7f5661cbb21866c068b2527826fcef9b3ca3fd0d1"} Nov 03 01:50:05.048821 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:05.048795 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8532fe271f091974ba738c7f5661cbb21866c068b2527826fcef9b3ca3fd0d1" Nov 03 01:50:05.049068 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:05.049050 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69306e62-dc8fc" Nov 03 01:50:11.431560 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.431529 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:50:11.443178 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.443147 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-cdfnj_7c5244e9-0e47-4a6d-a27d-24ce8455a537/console-operator/1.log" Nov 03 01:50:11.595424 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.595387 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq"] Nov 03 01:50:11.595942 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.595926 5144 cpu_manager.go:414] "containerMap: removing container" podUID="0f83fe52-aeb6-4b53-82aa-b3734985eb2f" containerName="revoke-keys" Nov 03 01:50:11.596005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.595943 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f83fe52-aeb6-4b53-82aa-b3734985eb2f" containerName="revoke-keys" Nov 03 01:50:11.596046 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.596026 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f83fe52-aeb6-4b53-82aa-b3734985eb2f" containerName="revoke-keys" Nov 03 01:50:11.599528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.599502 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.655945 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.655910 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq"] Nov 03 01:50:11.684610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.684523 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89wk9\" (UniqueName: \"kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.684610 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.684569 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.684841 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.684712 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.701016 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.700984 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:11.701205 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.701078 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:50:11.786038 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.786000 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89wk9\" (UniqueName: \"kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.786256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.786051 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.786256 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.786119 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.786831 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.786806 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.788822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.788799 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.842700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.842668 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89wk9\" (UniqueName: \"kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9\") pod \"maas-api-revoke-sub-6069f0d0-vc9kq\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:11.913394 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:11.913356 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:12.156547 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:50:12.156509 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:50:12.290487 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:12.290454 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq"] Nov 03 01:50:12.291821 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:50:12.291781 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf081827_3d44_406b_b6d4_dd8f54eab159.slice/crio-9c3284ba7d28a0607695ad723aaa7533a3c87d0b2642a0d185ecb67a00c962f8 WatchSource:0}: Error finding container 9c3284ba7d28a0607695ad723aaa7533a3c87d0b2642a0d185ecb67a00c962f8: Status 404 returned error can't find the container with id 9c3284ba7d28a0607695ad723aaa7533a3c87d0b2642a0d185ecb67a00c962f8 Nov 03 01:50:13.101166 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:13.101123 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" event={"ID":"bf081827-3d44-406b-b6d4-dd8f54eab159","Type":"ContainerStarted","Data":"22965b94c71f022991caf157eb9f7a71c11730e99dc44bdf3f218ce90be36229"} Nov 03 01:50:13.101721 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:13.101192 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" event={"ID":"bf081827-3d44-406b-b6d4-dd8f54eab159","Type":"ContainerStarted","Data":"9c3284ba7d28a0607695ad723aaa7533a3c87d0b2642a0d185ecb67a00c962f8"} Nov 03 01:50:13.213345 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:13.213285 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" podStartSLOduration=2.213268708 podStartE2EDuration="2.213268708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:13.212122085 +0000 UTC m=+6303.357816542" watchObservedRunningTime="2026-11-03 01:50:13.213268708 +0000 UTC m=+6303.358963165" Nov 03 01:50:14.110752 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:14.110718 5144 generic.go:356] "Generic (PLEG): container finished" podID="bf081827-3d44-406b-b6d4-dd8f54eab159" containerID="22965b94c71f022991caf157eb9f7a71c11730e99dc44bdf3f218ce90be36229" exitCode=0 Nov 03 01:50:14.111181 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:14.110807 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" event={"ID":"bf081827-3d44-406b-b6d4-dd8f54eab159","Type":"ContainerDied","Data":"22965b94c71f022991caf157eb9f7a71c11730e99dc44bdf3f218ce90be36229"} Nov 03 01:50:15.273396 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.273361 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:15.323682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.323626 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca\") pod \"bf081827-3d44-406b-b6d4-dd8f54eab159\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " Nov 03 01:50:15.323878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.323761 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca\") pod \"bf081827-3d44-406b-b6d4-dd8f54eab159\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " Nov 03 01:50:15.323925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.323897 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9\" (UniqueName: \"kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9\") pod \"bf081827-3d44-406b-b6d4-dd8f54eab159\" (UID: \"bf081827-3d44-406b-b6d4-dd8f54eab159\") " Nov 03 01:50:15.324098 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.324069 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca" pod "bf081827-3d44-406b-b6d4-dd8f54eab159" (UID: "bf081827-3d44-406b-b6d4-dd8f54eab159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:15.324240 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.324224 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:15.326152 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.326122 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9" pod "bf081827-3d44-406b-b6d4-dd8f54eab159" (UID: "bf081827-3d44-406b-b6d4-dd8f54eab159"). InnerVolumeSpecName "kube-api-access-89wk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:15.326293 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.326206 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca" pod "bf081827-3d44-406b-b6d4-dd8f54eab159" (UID: "bf081827-3d44-406b-b6d4-dd8f54eab159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:15.425505 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.425404 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf081827-3d44-406b-b6d4-dd8f54eab159-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:15.425505 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:15.425446 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89wk9\" (UniqueName: \"kubernetes.io/projected/bf081827-3d44-406b-b6d4-dd8f54eab159-kube-api-access-89wk9\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:16.130118 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:16.130075 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" Nov 03 01:50:16.130334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:16.130074 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6069f0d0-vc9kq" event={"ID":"bf081827-3d44-406b-b6d4-dd8f54eab159","Type":"ContainerDied","Data":"9c3284ba7d28a0607695ad723aaa7533a3c87d0b2642a0d185ecb67a00c962f8"} Nov 03 01:50:16.130334 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:16.130240 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c3284ba7d28a0607695ad723aaa7533a3c87d0b2642a0d185ecb67a00c962f8" Nov 03 01:50:22.289355 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.289323 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc"] Nov 03 01:50:22.289859 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.289844 5144 cpu_manager.go:414] "containerMap: removing container" podUID="bf081827-3d44-406b-b6d4-dd8f54eab159" containerName="revoke-keys" Nov 03 01:50:22.289910 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.289861 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf081827-3d44-406b-b6d4-dd8f54eab159" containerName="revoke-keys" Nov 03 01:50:22.289951 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.289940 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf081827-3d44-406b-b6d4-dd8f54eab159" containerName="revoke-keys" Nov 03 01:50:22.293279 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.293264 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.348208 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.348175 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc"] Nov 03 01:50:22.393376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.393335 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:22.393376 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.393369 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:50:22.435803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.435765 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.436037 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.435950 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prfhs\" (UniqueName: \"kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.436037 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.436001 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.537359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.537313 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prfhs\" (UniqueName: \"kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.537359 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.537366 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.537597 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.537407 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.538090 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.538070 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.540074 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.540019 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.594411 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.594376 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prfhs\" (UniqueName: \"kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs\") pod \"maas-api-revoke-sub-f063d6c0-m2pjc\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.607242 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.607213 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:22.982041 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:22.982010 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc"] Nov 03 01:50:22.983367 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:50:22.983340 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22761ee4_cd45_4b76_8c51_08a0859182d1.slice/crio-56af3b3eebd2fd337531a785772f87948fccf8f5a2b41b1a5a2ab32a4fcf1cfe WatchSource:0}: Error finding container 56af3b3eebd2fd337531a785772f87948fccf8f5a2b41b1a5a2ab32a4fcf1cfe: Status 404 returned error can't find the container with id 56af3b3eebd2fd337531a785772f87948fccf8f5a2b41b1a5a2ab32a4fcf1cfe Nov 03 01:50:23.211586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:23.211545 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" event={"ID":"22761ee4-cd45-4b76-8c51-08a0859182d1","Type":"ContainerStarted","Data":"56af3b3eebd2fd337531a785772f87948fccf8f5a2b41b1a5a2ab32a4fcf1cfe"} Nov 03 01:50:24.220939 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:24.220848 5144 generic.go:356] "Generic (PLEG): container finished" podID="22761ee4-cd45-4b76-8c51-08a0859182d1" containerID="413b09ffb974fcc65c2124e40cd54f8c73986589e568f1856f85fafea255dd12" exitCode=0 Nov 03 01:50:24.221325 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:24.220929 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" event={"ID":"22761ee4-cd45-4b76-8c51-08a0859182d1","Type":"ContainerDied","Data":"413b09ffb974fcc65c2124e40cd54f8c73986589e568f1856f85fafea255dd12"} Nov 03 01:50:25.370835 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.370811 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:25.569547 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.569517 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs\" (UniqueName: \"kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs\") pod \"22761ee4-cd45-4b76-8c51-08a0859182d1\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " Nov 03 01:50:25.569745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.569587 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca\") pod \"22761ee4-cd45-4b76-8c51-08a0859182d1\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " Nov 03 01:50:25.569745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.569713 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca\") pod \"22761ee4-cd45-4b76-8c51-08a0859182d1\" (UID: \"22761ee4-cd45-4b76-8c51-08a0859182d1\") " Nov 03 01:50:25.570075 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.570048 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca" pod "22761ee4-cd45-4b76-8c51-08a0859182d1" (UID: "22761ee4-cd45-4b76-8c51-08a0859182d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:25.571753 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.571732 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca" pod "22761ee4-cd45-4b76-8c51-08a0859182d1" (UID: "22761ee4-cd45-4b76-8c51-08a0859182d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:25.571852 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.571828 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs" pod "22761ee4-cd45-4b76-8c51-08a0859182d1" (UID: "22761ee4-cd45-4b76-8c51-08a0859182d1"). InnerVolumeSpecName "kube-api-access-prfhs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:25.670568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.670531 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prfhs\" (UniqueName: \"kubernetes.io/projected/22761ee4-cd45-4b76-8c51-08a0859182d1-kube-api-access-prfhs\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:25.670568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.670564 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:25.670568 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:25.670575 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/22761ee4-cd45-4b76-8c51-08a0859182d1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:26.234671 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:26.234626 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" Nov 03 01:50:26.234868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:26.234661 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f063d6c0-m2pjc" event={"ID":"22761ee4-cd45-4b76-8c51-08a0859182d1","Type":"ContainerDied","Data":"56af3b3eebd2fd337531a785772f87948fccf8f5a2b41b1a5a2ab32a4fcf1cfe"} Nov 03 01:50:26.234868 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:26.234718 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56af3b3eebd2fd337531a785772f87948fccf8f5a2b41b1a5a2ab32a4fcf1cfe" Nov 03 01:50:33.046261 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.046225 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24"] Nov 03 01:50:33.046726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.046702 5144 cpu_manager.go:414] "containerMap: removing container" podUID="22761ee4-cd45-4b76-8c51-08a0859182d1" containerName="revoke-keys" Nov 03 01:50:33.046726 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.046715 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22761ee4-cd45-4b76-8c51-08a0859182d1" containerName="revoke-keys" Nov 03 01:50:33.046822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.046798 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="22761ee4-cd45-4b76-8c51-08a0859182d1" containerName="revoke-keys" Nov 03 01:50:33.050136 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.050093 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.131552 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.131514 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:33.131803 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.131679 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:50:33.135007 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.134984 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24"] Nov 03 01:50:33.138101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.138082 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.138220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.138198 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gml6p\" (UniqueName: \"kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.138289 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.138278 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.239777 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.239742 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gml6p\" (UniqueName: \"kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.239964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.239817 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.239964 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.239874 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.240621 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.240600 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.242482 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.242457 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.296064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.296034 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gml6p\" (UniqueName: \"kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p\") pod \"maas-api-revoke-sub-9583bd7c-gqt24\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.363288 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.363200 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:33.741059 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:33.741027 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24"] Nov 03 01:50:33.742126 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:50:33.742091 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325e2061_3552_4439_9784_98ebcaa24f53.slice/crio-abcbcb78f30a7570add3e84ceb95918f339a457334b4cbef3ff5a7e47e749e18 WatchSource:0}: Error finding container abcbcb78f30a7570add3e84ceb95918f339a457334b4cbef3ff5a7e47e749e18: Status 404 returned error can't find the container with id abcbcb78f30a7570add3e84ceb95918f339a457334b4cbef3ff5a7e47e749e18 Nov 03 01:50:34.289323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:34.289229 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" event={"ID":"325e2061-3552-4439-9784-98ebcaa24f53","Type":"ContainerStarted","Data":"d449d854fc07cbd1e2459197250bc6a4552e2588bd0fee4930ab29197a7463b5"} Nov 03 01:50:34.289323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:34.289298 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" event={"ID":"325e2061-3552-4439-9784-98ebcaa24f53","Type":"ContainerStarted","Data":"abcbcb78f30a7570add3e84ceb95918f339a457334b4cbef3ff5a7e47e749e18"} Nov 03 01:50:34.406814 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:34.406756 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" podStartSLOduration=1.4067406 podStartE2EDuration="1.4067406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:34.405474046 +0000 UTC m=+6324.551168519" watchObservedRunningTime="2026-11-03 01:50:34.4067406 +0000 UTC m=+6324.552435056" Nov 03 01:50:35.297290 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:35.297252 5144 generic.go:356] "Generic (PLEG): container finished" podID="325e2061-3552-4439-9784-98ebcaa24f53" containerID="d449d854fc07cbd1e2459197250bc6a4552e2588bd0fee4930ab29197a7463b5" exitCode=0 Nov 03 01:50:35.297827 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:35.297329 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" event={"ID":"325e2061-3552-4439-9784-98ebcaa24f53","Type":"ContainerDied","Data":"d449d854fc07cbd1e2459197250bc6a4552e2588bd0fee4930ab29197a7463b5"} Nov 03 01:50:36.465315 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.465280 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:36.593983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.593027 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca\") pod \"325e2061-3552-4439-9784-98ebcaa24f53\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " Nov 03 01:50:36.593983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.593235 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p\" (UniqueName: \"kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p\") pod \"325e2061-3552-4439-9784-98ebcaa24f53\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " Nov 03 01:50:36.593983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.593295 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca\") pod \"325e2061-3552-4439-9784-98ebcaa24f53\" (UID: \"325e2061-3552-4439-9784-98ebcaa24f53\") " Nov 03 01:50:36.594857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.594501 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca" pod "325e2061-3552-4439-9784-98ebcaa24f53" (UID: "325e2061-3552-4439-9784-98ebcaa24f53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:36.596950 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.596917 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca" pod "325e2061-3552-4439-9784-98ebcaa24f53" (UID: "325e2061-3552-4439-9784-98ebcaa24f53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:36.597430 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.597386 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p" pod "325e2061-3552-4439-9784-98ebcaa24f53" (UID: "325e2061-3552-4439-9784-98ebcaa24f53"). InnerVolumeSpecName "kube-api-access-gml6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:36.695064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.694954 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gml6p\" (UniqueName: \"kubernetes.io/projected/325e2061-3552-4439-9784-98ebcaa24f53-kube-api-access-gml6p\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:36.695064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.694997 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:36.695064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:36.695034 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325e2061-3552-4439-9784-98ebcaa24f53-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:37.313702 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:37.313671 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" Nov 03 01:50:37.314060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:37.313697 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9583bd7c-gqt24" event={"ID":"325e2061-3552-4439-9784-98ebcaa24f53","Type":"ContainerDied","Data":"abcbcb78f30a7570add3e84ceb95918f339a457334b4cbef3ff5a7e47e749e18"} Nov 03 01:50:37.314060 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:37.314033 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="abcbcb78f30a7570add3e84ceb95918f339a457334b4cbef3ff5a7e47e749e18" Nov 03 01:50:38.061743 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:38.061696 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn"] Nov 03 01:50:38.065220 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:38.065165 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b39b932-fc5qn"] Nov 03 01:50:39.995360 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:39.995320 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="880843af-cb33-4198-b699-22fd72d9b133" path="/var/lib/kubelet/pods/880843af-cb33-4198-b699-22fd72d9b133/volumes" Nov 03 01:50:42.157254 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:50:42.157218 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:50:43.874519 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.874479 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n"] Nov 03 01:50:43.875037 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.874982 5144 cpu_manager.go:414] "containerMap: removing container" podUID="325e2061-3552-4439-9784-98ebcaa24f53" containerName="revoke-keys" Nov 03 01:50:43.875037 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.874996 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325e2061-3552-4439-9784-98ebcaa24f53" containerName="revoke-keys" Nov 03 01:50:43.875135 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.875073 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="325e2061-3552-4439-9784-98ebcaa24f53" containerName="revoke-keys" Nov 03 01:50:43.878774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.878746 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:43.934948 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.934898 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n"] Nov 03 01:50:43.971026 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.970984 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:43.971248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.971048 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p47jp\" (UniqueName: \"kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:43.971248 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.971156 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:43.980148 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.980115 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:43.980294 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:43.980198 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:50:44.072072 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.072026 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.072311 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.072230 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.072383 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.072325 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p47jp\" (UniqueName: \"kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.073025 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.072997 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.074689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.074667 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.129521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.129439 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p47jp\" (UniqueName: \"kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp\") pod \"maas-api-revoke-sub-ec0c397f-s8h5n\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.193111 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.193067 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:44.573022 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:44.572993 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n"] Nov 03 01:50:44.573412 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:50:44.573386 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339f8e95_22ec_4c0a_9cd6_eca9501d5403.slice/crio-fa5b46bd0b9148bc70dc62ff3d5b14677cdbb97414b6a2b192619f2842821c88 WatchSource:0}: Error finding container fa5b46bd0b9148bc70dc62ff3d5b14677cdbb97414b6a2b192619f2842821c88: Status 404 returned error can't find the container with id fa5b46bd0b9148bc70dc62ff3d5b14677cdbb97414b6a2b192619f2842821c88 Nov 03 01:50:45.367507 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:45.367467 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" event={"ID":"339f8e95-22ec-4c0a-9cd6-eca9501d5403","Type":"ContainerStarted","Data":"51f9a73075a3bb48ecdbde5a98e30ea53b68d259db5945680c9122a7cbbf9c26"} Nov 03 01:50:45.367994 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:45.367538 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" event={"ID":"339f8e95-22ec-4c0a-9cd6-eca9501d5403","Type":"ContainerStarted","Data":"fa5b46bd0b9148bc70dc62ff3d5b14677cdbb97414b6a2b192619f2842821c88"} Nov 03 01:50:45.480980 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:45.480924 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" podStartSLOduration=2.480905789 podStartE2EDuration="2.480905789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:50:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:50:45.479451794 +0000 UTC m=+6335.625146269" watchObservedRunningTime="2026-11-03 01:50:45.480905789 +0000 UTC m=+6335.626600247" Nov 03 01:50:46.375513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:46.375472 5144 generic.go:356] "Generic (PLEG): container finished" podID="339f8e95-22ec-4c0a-9cd6-eca9501d5403" containerID="51f9a73075a3bb48ecdbde5a98e30ea53b68d259db5945680c9122a7cbbf9c26" exitCode=0 Nov 03 01:50:46.375992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:46.375560 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" event={"ID":"339f8e95-22ec-4c0a-9cd6-eca9501d5403","Type":"ContainerDied","Data":"51f9a73075a3bb48ecdbde5a98e30ea53b68d259db5945680c9122a7cbbf9c26"} Nov 03 01:50:47.861672 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.836333 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:47.969664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.969029 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca\") pod \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " Nov 03 01:50:47.969664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.969114 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca\") pod \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " Nov 03 01:50:47.969664 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.969164 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp\" (UniqueName: \"kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp\") pod \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\" (UID: \"339f8e95-22ec-4c0a-9cd6-eca9501d5403\") " Nov 03 01:50:47.974366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.974313 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca" pod "339f8e95-22ec-4c0a-9cd6-eca9501d5403" (UID: "339f8e95-22ec-4c0a-9cd6-eca9501d5403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:47.998772 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.998724 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca" pod "339f8e95-22ec-4c0a-9cd6-eca9501d5403" (UID: "339f8e95-22ec-4c0a-9cd6-eca9501d5403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:47.998975 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:47.998855 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp" pod "339f8e95-22ec-4c0a-9cd6-eca9501d5403" (UID: "339f8e95-22ec-4c0a-9cd6-eca9501d5403"). InnerVolumeSpecName "kube-api-access-p47jp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:48.085731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:48.075406 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:48.085731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:48.075453 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/339f8e95-22ec-4c0a-9cd6-eca9501d5403-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:48.085731 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:48.075471 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p47jp\" (UniqueName: \"kubernetes.io/projected/339f8e95-22ec-4c0a-9cd6-eca9501d5403-kube-api-access-p47jp\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:48.403918 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:48.403806 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" event={"ID":"339f8e95-22ec-4c0a-9cd6-eca9501d5403","Type":"ContainerDied","Data":"fa5b46bd0b9148bc70dc62ff3d5b14677cdbb97414b6a2b192619f2842821c88"} Nov 03 01:50:48.403918 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:48.403880 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa5b46bd0b9148bc70dc62ff3d5b14677cdbb97414b6a2b192619f2842821c88" Nov 03 01:50:48.403918 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:48.403892 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec0c397f-s8h5n" Nov 03 01:50:49.052906 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:49.052622 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6"] Nov 03 01:50:49.058967 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:49.058926 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc82e4a3-2w4s6"] Nov 03 01:50:49.996366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:49.996331 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="655f05f4-ebf1-4cca-bcb3-39fcd65d94d9" path="/var/lib/kubelet/pods/655f05f4-ebf1-4cca-bcb3-39fcd65d94d9/volumes" Nov 03 01:50:54.688214 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.688181 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7"] Nov 03 01:50:54.688738 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.688723 5144 cpu_manager.go:414] "containerMap: removing container" podUID="339f8e95-22ec-4c0a-9cd6-eca9501d5403" containerName="revoke-keys" Nov 03 01:50:54.688793 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.688740 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="339f8e95-22ec-4c0a-9cd6-eca9501d5403" containerName="revoke-keys" Nov 03 01:50:54.688851 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.688841 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="339f8e95-22ec-4c0a-9cd6-eca9501d5403" containerName="revoke-keys" Nov 03 01:50:54.692532 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.692513 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.748687 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.748654 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7"] Nov 03 01:50:54.755462 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.755418 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.755662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.755485 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.755662 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.755516 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt4pd\" (UniqueName: \"kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.793802 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.793759 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:50:54.793989 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.793831 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:50:54.856058 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.856011 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.856281 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.856070 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.856281 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.856120 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt4pd\" (UniqueName: \"kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.856882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.856856 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.858825 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.858799 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:54.912857 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:54.912827 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt4pd\" (UniqueName: \"kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd\") pod \"maas-api-revoke-sub-2259f8a5-mnqn7\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:55.006658 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:55.006597 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:55.383666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:55.383618 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7"] Nov 03 01:50:55.384334 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:50:55.384301 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1433b9d6_361a_498b_a0f5_e2f318b1b1f1.slice/crio-176cada1006d1828b6533219de191baaabdfdc4483f08d9a39ccf2eae9c5aeaa WatchSource:0}: Error finding container 176cada1006d1828b6533219de191baaabdfdc4483f08d9a39ccf2eae9c5aeaa: Status 404 returned error can't find the container with id 176cada1006d1828b6533219de191baaabdfdc4483f08d9a39ccf2eae9c5aeaa Nov 03 01:50:55.448399 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:55.448362 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" event={"ID":"1433b9d6-361a-498b-a0f5-e2f318b1b1f1","Type":"ContainerStarted","Data":"176cada1006d1828b6533219de191baaabdfdc4483f08d9a39ccf2eae9c5aeaa"} Nov 03 01:50:56.456665 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:56.456612 5144 generic.go:356] "Generic (PLEG): container finished" podID="1433b9d6-361a-498b-a0f5-e2f318b1b1f1" containerID="ef745f0ebff72775b8e24cc68c67e20dc30757a1aca8a0541cde5b725579958c" exitCode=0 Nov 03 01:50:56.457188 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:56.456708 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" event={"ID":"1433b9d6-361a-498b-a0f5-e2f318b1b1f1","Type":"ContainerDied","Data":"ef745f0ebff72775b8e24cc68c67e20dc30757a1aca8a0541cde5b725579958c"} Nov 03 01:50:57.645822 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.645783 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:57.689770 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.689717 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca\") pod \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " Nov 03 01:50:57.689992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.689851 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca\") pod \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " Nov 03 01:50:57.689992 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.689904 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd\" (UniqueName: \"kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd\") pod \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\" (UID: \"1433b9d6-361a-498b-a0f5-e2f318b1b1f1\") " Nov 03 01:50:57.690322 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.690284 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca" pod "1433b9d6-361a-498b-a0f5-e2f318b1b1f1" (UID: "1433b9d6-361a-498b-a0f5-e2f318b1b1f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:50:57.692267 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.692231 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca" pod "1433b9d6-361a-498b-a0f5-e2f318b1b1f1" (UID: "1433b9d6-361a-498b-a0f5-e2f318b1b1f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:50:57.692956 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.692934 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd" pod "1433b9d6-361a-498b-a0f5-e2f318b1b1f1" (UID: "1433b9d6-361a-498b-a0f5-e2f318b1b1f1"). InnerVolumeSpecName "kube-api-access-dt4pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:50:57.791064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.791016 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:57.791064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.791060 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:57.791377 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:57.791077 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt4pd\" (UniqueName: \"kubernetes.io/projected/1433b9d6-361a-498b-a0f5-e2f318b1b1f1-kube-api-access-dt4pd\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:50:58.487469 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:58.487433 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" Nov 03 01:50:58.488535 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:58.487546 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2259f8a5-mnqn7" event={"ID":"1433b9d6-361a-498b-a0f5-e2f318b1b1f1","Type":"ContainerDied","Data":"176cada1006d1828b6533219de191baaabdfdc4483f08d9a39ccf2eae9c5aeaa"} Nov 03 01:50:58.488774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:50:58.488756 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="176cada1006d1828b6533219de191baaabdfdc4483f08d9a39ccf2eae9c5aeaa" Nov 03 01:51:05.485764 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.485724 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq"] Nov 03 01:51:05.488421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.486411 5144 cpu_manager.go:414] "containerMap: removing container" podUID="1433b9d6-361a-498b-a0f5-e2f318b1b1f1" containerName="revoke-keys" Nov 03 01:51:05.488421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.486428 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1433b9d6-361a-498b-a0f5-e2f318b1b1f1" containerName="revoke-keys" Nov 03 01:51:05.488421 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.486521 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="1433b9d6-361a-498b-a0f5-e2f318b1b1f1" containerName="revoke-keys" Nov 03 01:51:05.489666 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.489619 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.548902 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.548859 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq"] Nov 03 01:51:05.592694 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.592655 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 01:51:05.592694 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.592662 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-55jkx\"" Nov 03 01:51:05.600194 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.600151 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qj95h\" (UniqueName: \"kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.600388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.600310 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.600388 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.600368 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.701009 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.700977 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qj95h\" (UniqueName: \"kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.701234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.701048 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.701234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.701093 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.701842 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.701816 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.703781 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.703755 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.757952 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.757915 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qj95h\" (UniqueName: \"kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h\") pod \"maas-api-revoke-sub-ff50e569-jhwcq\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:05.806914 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:05.806873 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:06.187689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:06.187659 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq"] Nov 03 01:51:06.188917 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:51:06.188873 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95b5c249_e2af_4236_899c_2753ca6a4ed0.slice/crio-700a083623fc991b8e1ff4df625b0843f257f06c1210efffd2a8f3a9513e3fae WatchSource:0}: Error finding container 700a083623fc991b8e1ff4df625b0843f257f06c1210efffd2a8f3a9513e3fae: Status 404 returned error can't find the container with id 700a083623fc991b8e1ff4df625b0843f257f06c1210efffd2a8f3a9513e3fae Nov 03 01:51:06.546406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:06.546372 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" event={"ID":"95b5c249-e2af-4236-899c-2753ca6a4ed0","Type":"ContainerStarted","Data":"709f12bb7214c833e06309def073080570fd534496b775bf50851c8b89ba0425"} Nov 03 01:51:06.546882 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:06.546434 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" event={"ID":"95b5c249-e2af-4236-899c-2753ca6a4ed0","Type":"ContainerStarted","Data":"700a083623fc991b8e1ff4df625b0843f257f06c1210efffd2a8f3a9513e3fae"} Nov 03 01:51:06.661682 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:06.661608 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" podStartSLOduration=1.66159304 podStartE2EDuration="1.66159304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:51:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:51:06.658020947 +0000 UTC m=+6356.803715404" watchObservedRunningTime="2026-11-03 01:51:06.66159304 +0000 UTC m=+6356.807287497" Nov 03 01:51:07.554189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:07.554086 5144 generic.go:356] "Generic (PLEG): container finished" podID="95b5c249-e2af-4236-899c-2753ca6a4ed0" containerID="709f12bb7214c833e06309def073080570fd534496b775bf50851c8b89ba0425" exitCode=0 Nov 03 01:51:07.554189 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:07.554174 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" event={"ID":"95b5c249-e2af-4236-899c-2753ca6a4ed0","Type":"ContainerDied","Data":"709f12bb7214c833e06309def073080570fd534496b775bf50851c8b89ba0425"} Nov 03 01:51:09.013653 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.013602 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:09.201660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.201373 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca\") pod \"95b5c249-e2af-4236-899c-2753ca6a4ed0\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " Nov 03 01:51:09.201660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.201473 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h\" (UniqueName: \"kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h\") pod \"95b5c249-e2af-4236-899c-2753ca6a4ed0\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " Nov 03 01:51:09.201660 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.201531 5144 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca\") pod \"95b5c249-e2af-4236-899c-2753ca6a4ed0\" (UID: \"95b5c249-e2af-4236-899c-2753ca6a4ed0\") " Nov 03 01:51:09.206064 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.202409 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca" pod "95b5c249-e2af-4236-899c-2753ca6a4ed0" (UID: "95b5c249-e2af-4236-899c-2753ca6a4ed0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 01:51:09.212110 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.212071 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca" pod "95b5c249-e2af-4236-899c-2753ca6a4ed0" (UID: "95b5c249-e2af-4236-899c-2753ca6a4ed0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 01:51:09.213714 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.213202 5144 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h" pod "95b5c249-e2af-4236-899c-2753ca6a4ed0" (UID: "95b5c249-e2af-4236-899c-2753ca6a4ed0"). InnerVolumeSpecName "kube-api-access-qj95h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 01:51:09.303433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.303393 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-xks-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:51:09.303433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.303427 5144 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qj95h\" (UniqueName: \"kubernetes.io/projected/95b5c249-e2af-4236-899c-2753ca6a4ed0-kube-api-access-qj95h\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:51:09.303433 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.303439 5144 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95b5c249-e2af-4236-899c-2753ca6a4ed0-maas-api-openshift-ca\") on node \"ip-10-0-143-162.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 01:51:09.603459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.603426 5144 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" Nov 03 01:51:09.603459 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.603441 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ff50e569-jhwcq" event={"ID":"95b5c249-e2af-4236-899c-2753ca6a4ed0","Type":"ContainerDied","Data":"700a083623fc991b8e1ff4df625b0843f257f06c1210efffd2a8f3a9513e3fae"} Nov 03 01:51:09.603725 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:09.603489 5144 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="700a083623fc991b8e1ff4df625b0843f257f06c1210efffd2a8f3a9513e3fae" Nov 03 01:51:12.156196 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:51:12.156152 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:51:15.260681 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:15.260649 5144 scope.go:122] "RemoveContainer" containerID="2b60e6f6a1b96efca74a7144dfae6758803b830aad043bf37baaf389f4f1ca36" Nov 03 01:51:15.270904 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:15.270883 5144 scope.go:122] "RemoveContainer" containerID="1ee4da98d2365356d48a3465fb450dbe234f55fbb0ee642c3e00309317bdeb0a" Nov 03 01:51:20.035371 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:20.035327 5144 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 01:51:20.036034 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:20.036000 5144 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq" podUID="9ea14538-faad-45d7-b7b8-8ac4e0362b0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 01:51:24.048774 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:24.048736 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47"] Nov 03 01:51:24.075624 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:24.075584 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-669bbc7b-4mn47"] Nov 03 01:51:25.995084 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:25.995041 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68ca6874-189d-4ab7-8421-0ae0122caed3" path="/var/lib/kubelet/pods/68ca6874-189d-4ab7-8421-0ae0122caed3/volumes" Nov 03 01:51:37.502348 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:37.502310 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64cfdbc956-4ltf2_a2768863-e2e0-46bd-9369-58c6a8a140d6/authorino/0.log" Nov 03 01:51:42.165011 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:51:42.164964 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:51:49.704856 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:49.704807 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67cc775cd9-n49hj_af51a4fd-3ae1-46ac-b602-208c8a54c058/manager/0.log" Nov 03 01:51:51.290570 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:51.290535 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785d4477d7-2j7w9_8843a38e-7855-4a1c-88c1-113c9c723ff5/manager/0.log" Nov 03 01:51:51.748259 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:51.748168 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8q225_2dc160d5-869c-4f83-9abc-0f3a0018740f/server/0.log" Nov 03 01:51:52.250598 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:52.250561 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-m2z5c_916b609e-7a1f-4137-83b5-424605bc72b0/proxy/0.log" Nov 03 01:51:53.874428 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:53.874375 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2259f8a5-mnqn7_1433b9d6-361a-498b-a0f5-e2f318b1b1f1/revoke-keys/0.log" Nov 03 01:51:54.095981 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:54.095948 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6069f0d0-vc9kq_bf081827-3d44-406b-b6d4-dd8f54eab159/revoke-keys/0.log" Nov 03 01:51:54.321987 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:54.321957 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-69306e62-dc8fc_0f83fe52-aeb6-4b53-82aa-b3734985eb2f/revoke-keys/0.log" Nov 03 01:51:54.546556 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:54.546528 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-77111d9f-m6525_eb1bc4a8-227f-408a-a2cb-de3b952622d5/revoke-keys/0.log" Nov 03 01:51:54.769834 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:54.769802 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-9583bd7c-gqt24_325e2061-3552-4439-9784-98ebcaa24f53/revoke-keys/0.log" Nov 03 01:51:55.002005 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:55.001968 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a211ca3a-vw4fz_3d1331ac-1173-44a5-b7e5-06f314e938b1/revoke-keys/0.log" Nov 03 01:51:55.229955 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:55.229868 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-b3655629-4n7hp_58a715a7-85d4-4567-b549-12daf9b79983/revoke-keys/0.log" Nov 03 01:51:55.452295 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:55.452264 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ced98d90-2mspz_417201ed-aeba-46ec-acb2-281076d53f6f/revoke-keys/0.log" Nov 03 01:51:55.679269 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:55.679241 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2614c30-f7d6p_eb362b89-1a32-4f21-a6a8-52af65337218/revoke-keys/0.log" Nov 03 01:51:55.891206 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:55.891175 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d3b8d90c-qkbb6_2e2bb2fd-c4fc-4e36-9a6b-5b4242d1382c/revoke-keys/0.log" Nov 03 01:51:56.120104 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:56.120073 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-dbe653a3-72l7r_720fc652-4219-49c4-9156-ce5ddff6d188/revoke-keys/0.log" Nov 03 01:51:56.349792 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:56.349760 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e3d58e9d-krvk4_09456d5f-e8fe-432a-b11f-093f94a8a8e7/revoke-keys/0.log" Nov 03 01:51:56.586461 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:56.586425 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ec0c397f-s8h5n_339f8e95-22ec-4c0a-9cd6-eca9501d5403/revoke-keys/0.log" Nov 03 01:51:56.818693 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:56.818625 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f063d6c0-m2pjc_22761ee4-cd45-4b76-8c51-08a0859182d1/revoke-keys/0.log" Nov 03 01:51:57.054830 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:57.054799 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f341e88a-x554v_00f8ca20-c82c-414f-9ea8-658d3c692053/revoke-keys/0.log" Nov 03 01:51:57.284165 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:57.284133 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-ff50e569-jhwcq_95b5c249-e2af-4236-899c-2753ca6a4ed0/revoke-keys/0.log" Nov 03 01:51:57.512235 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:57.512200 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-pgff6_57f5d573-36e8-4eed-a770-e43e96ee61b4/postgres/0.log" Nov 03 01:51:59.589970 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:51:59.589940 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64cfdbc956-4ltf2_a2768863-e2e0-46bd-9369-58c6a8a140d6/authorino/0.log" Nov 03 01:52:00.919982 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:00.919860 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-qx2hg_3ae333cb-5f6f-437f-9ba6-5cb249330692/manager/0.log" Nov 03 01:52:02.457141 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:02.457107 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj_e01a5e11-095c-402f-bfeb-07032cc4455c/istio-proxy/0.log" Nov 03 01:52:03.051984 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:03.051951 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54744779bc-6nckb_9b8e9601-f07e-4271-8d87-a6bfbbe91d9d/kube-auth-proxy/0.log" Nov 03 01:52:03.487591 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:03.487496 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs_c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14/istio-proxy/0.log" Nov 03 01:52:04.318925 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:04.318887 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq_9ea14538-faad-45d7-b7b8-8ac4e0362b0d/istio-proxy/0.log" Nov 03 01:52:05.177323 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:05.177289 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw_43f74203-a9e7-4cfe-ab2a-596a0ea37e1c/istio-proxy/0.log" Nov 03 01:52:06.304139 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:06.304105 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4_c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa/istio-proxy/0.log" Nov 03 01:52:06.807297 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:06.807262 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-ccd6f9449-lp87k_5e2b6a92-c4bb-40a2-a630-ae33f705e854/payload-processing/0.log" Nov 03 01:52:07.038780 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:07.038748 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767574d6c-vss4j_31e371fb-f718-43d0-bf0f-d909838ac2d0/router/0.log" Nov 03 01:52:08.692689 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:08.692651 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-kqqws_f1b7db33-4fcc-421a-b0f4-5a14c5da0312/main/0.log" Nov 03 01:52:08.963674 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:08.963569 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-k4hrj_c8dd533d-25a0-4034-bfbe-c9f9808a5cce/main/0.log" Nov 03 01:52:09.044853 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:09.044817 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525"] Nov 03 01:52:09.094156 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:09.094122 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77111d9f-m6525"] Nov 03 01:52:09.420947 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:09.420917 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-tsv68_83150290-61b8-4483-8c83-ecef4a633e4b/main/0.log" Nov 03 01:52:09.664480 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:09.664452 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-nfwcc_5b0129e1-2867-42f9-862b-96bf78fe011d/main/0.log" Nov 03 01:52:09.995778 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:09.995747 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb1bc4a8-227f-408a-a2cb-de3b952622d5" path="/var/lib/kubelet/pods/eb1bc4a8-227f-408a-a2cb-de3b952622d5/volumes" Nov 03 01:52:12.155735 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:52:12.155701 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:52:15.301406 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:15.301368 5144 scope.go:122] "RemoveContainer" containerID="182f95d57edf821804eedc6e2744210d331dfb2e2ae9efc29b45343cc48db27d" Nov 03 01:52:15.311473 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:15.311449 5144 scope.go:122] "RemoveContainer" containerID="31cb745c302e28b645b793f9c4a125b5b8e8a2a307261a367fd6f55f7786fb7a" Nov 03 01:52:19.044647 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:19.044553 5144 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4"] Nov 03 01:52:19.094140 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:19.094112 5144 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e3d58e9d-krvk4"] Nov 03 01:52:19.994900 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:19.994857 5144 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09456d5f-e8fe-432a-b11f-093f94a8a8e7" path="/var/lib/kubelet/pods/09456d5f-e8fe-432a-b11f-093f94a8a8e7/volumes" Nov 03 01:52:21.113042 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.113008 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-rd684/must-gather-7d7nf"] Nov 03 01:52:21.113901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.113876 5144 cpu_manager.go:414] "containerMap: removing container" podUID="95b5c249-e2af-4236-899c-2753ca6a4ed0" containerName="revoke-keys" Nov 03 01:52:21.113901 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.113900 5144 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95b5c249-e2af-4236-899c-2753ca6a4ed0" containerName="revoke-keys" Nov 03 01:52:21.114101 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.114006 5144 memory_manager.go:365] "RemoveStaleState removing state" podUID="95b5c249-e2af-4236-899c-2753ca6a4ed0" containerName="revoke-keys" Nov 03 01:52:21.119088 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.119065 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.172933 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.172893 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd684/must-gather-7d7nf"] Nov 03 01:52:21.190366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.190331 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pq95d\" (UniqueName: \"kubernetes.io/projected/e558e085-6e88-415e-b5a4-7d1eb1883d2c-kube-api-access-pq95d\") pod \"must-gather-7d7nf\" (UID: \"e558e085-6e88-415e-b5a4-7d1eb1883d2c\") " pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.190366 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.190367 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e558e085-6e88-415e-b5a4-7d1eb1883d2c-must-gather-output\") pod \"must-gather-7d7nf\" (UID: \"e558e085-6e88-415e-b5a4-7d1eb1883d2c\") " pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.218775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.218736 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd684\"/\"openshift-service-ca.crt\"" Nov 03 01:52:21.218775 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.218754 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-rd684\"/\"kube-root-ca.crt\"" Nov 03 01:52:21.219002 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.218818 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-rd684\"/\"default-dockercfg-slq9f\"" Nov 03 01:52:21.291745 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.291706 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pq95d\" (UniqueName: \"kubernetes.io/projected/e558e085-6e88-415e-b5a4-7d1eb1883d2c-kube-api-access-pq95d\") pod \"must-gather-7d7nf\" (UID: \"e558e085-6e88-415e-b5a4-7d1eb1883d2c\") " pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.291983 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.291753 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e558e085-6e88-415e-b5a4-7d1eb1883d2c-must-gather-output\") pod \"must-gather-7d7nf\" (UID: \"e558e085-6e88-415e-b5a4-7d1eb1883d2c\") " pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.292155 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.292134 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/e558e085-6e88-415e-b5a4-7d1eb1883d2c-must-gather-output\") pod \"must-gather-7d7nf\" (UID: \"e558e085-6e88-415e-b5a4-7d1eb1883d2c\") " pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.348790 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.348749 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pq95d\" (UniqueName: \"kubernetes.io/projected/e558e085-6e88-415e-b5a4-7d1eb1883d2c-kube-api-access-pq95d\") pod \"must-gather-7d7nf\" (UID: \"e558e085-6e88-415e-b5a4-7d1eb1883d2c\") " pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.434175 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.434088 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-rd684/must-gather-7d7nf" Nov 03 01:52:21.811409 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.811383 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-rd684/must-gather-7d7nf"] Nov 03 01:52:21.813556 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:52:21.813530 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode558e085_6e88_415e_b5a4_7d1eb1883d2c.slice/crio-6693c5a93ec9666f431eec16b9029f7be05c6158e8e740bb3248f6a316b06938 WatchSource:0}: Error finding container 6693c5a93ec9666f431eec16b9029f7be05c6158e8e740bb3248f6a316b06938: Status 404 returned error can't find the container with id 6693c5a93ec9666f431eec16b9029f7be05c6158e8e740bb3248f6a316b06938 Nov 03 01:52:21.815500 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:21.815479 5144 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 01:52:22.118979 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:22.118889 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd684/must-gather-7d7nf" event={"ID":"e558e085-6e88-415e-b5a4-7d1eb1883d2c","Type":"ContainerStarted","Data":"6693c5a93ec9666f431eec16b9029f7be05c6158e8e740bb3248f6a316b06938"} Nov 03 01:52:29.171318 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:29.171280 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd684/must-gather-7d7nf" event={"ID":"e558e085-6e88-415e-b5a4-7d1eb1883d2c","Type":"ContainerStarted","Data":"4cbfdfba18aa699ab2b64235837c8bbca1f77d710a2af9008ad614346bfd3efd"} Nov 03 01:52:29.171800 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:29.171325 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-rd684/must-gather-7d7nf" event={"ID":"e558e085-6e88-415e-b5a4-7d1eb1883d2c","Type":"ContainerStarted","Data":"5f664c467a586b98568807f9964f3425866fcd488a76c2fe933df73ab0d43b7c"} Nov 03 01:52:29.283926 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:29.283872 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-rd684/must-gather-7d7nf" podStartSLOduration=1.668408366 podStartE2EDuration="8.283856476s" totalImagesPullingTime="6.61544811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:52:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:29.282125133 +0000 UTC m=+6439.427819590" watchObservedRunningTime="2026-11-03 01:52:29.283856476 +0000 UTC m=+6439.429550961" Nov 03 01:52:30.490603 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:30.490569 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cw8kk_2ee7c64c-4945-469a-8d27-cfb702aabf50/global-pull-secret-syncer/0.log" Nov 03 01:52:31.662694 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:31.662659 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-gtptx_35ce84ec-29fd-49ab-9c7a-092e99f093b8/connection-checker/0.log" Nov 03 01:52:32.078912 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:32.078881 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-kzvvs_d7e0f3c7-a695-4b00-8178-1955b1159469/connection-checker/0.log" Nov 03 01:52:32.505369 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:32.505343 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-mvcq4_1edd18e2-9aa6-4708-98e9-773c0a38a368/connection-checker/0.log" Nov 03 01:52:32.730380 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:32.730340 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67cc775cd9-n49hj_af51a4fd-3ae1-46ac-b602-208c8a54c058/manager/0.log" Nov 03 01:52:34.018673 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:34.018621 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-z8d7r_1a0eb70c-74c9-485b-a907-fa2830431a1d/konnectivity-agent/0.log" Nov 03 01:52:34.345464 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:34.345388 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785d4477d7-2j7w9_8843a38e-7855-4a1c-88c1-113c9c723ff5/manager/0.log" Nov 03 01:52:34.454586 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:34.454555 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8q225_2dc160d5-869c-4f83-9abc-0f3a0018740f/server/0.log" Nov 03 01:52:35.248528 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:35.248487 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-143-162.us-west-1.compute.internal_6b8cde6519a41e629e9c7fa53310b7cb/haproxy/0.log" Nov 03 01:52:35.657233 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:35.657196 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-m2z5c_916b609e-7a1f-4137-83b5-424605bc72b0/proxy/0.log" Nov 03 01:52:38.575422 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:38.575386 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67cc775cd9-n49hj_af51a4fd-3ae1-46ac-b602-208c8a54c058/manager/0.log" Nov 03 01:52:39.384957 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.384916 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64cfdbc956-4ltf2_a2768863-e2e0-46bd-9369-58c6a8a140d6/authorino/0.log" Nov 03 01:52:39.800700 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.800663 5144 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh"] Nov 03 01:52:39.807870 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.807842 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.860142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.860107 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26269\"/\"kube-root-ca.crt\"" Nov 03 01:52:39.860142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.860107 5144 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26269\"/\"default-dockercfg-bj4wk\"" Nov 03 01:52:39.860142 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.860107 5144 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26269\"/\"openshift-service-ca.crt\"" Nov 03 01:52:39.895895 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.895850 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-lib-modules\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.896105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.896005 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-podres\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.896105 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.896031 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-sys\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.896234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.896115 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtd2l\" (UniqueName: \"kubernetes.io/projected/e292a072-7188-4a21-960f-ef5354c871c8-kube-api-access-qtd2l\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.896234 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.896202 5144 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-proc\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.916521 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.916474 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh"] Nov 03 01:52:39.997501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997467 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-podres\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997501 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997502 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-sys\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997543 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtd2l\" (UniqueName: \"kubernetes.io/projected/e292a072-7188-4a21-960f-ef5354c871c8-kube-api-access-qtd2l\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997597 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-proc\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997603 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-podres\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997607 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-sys\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997711 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-proc\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.997799 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997783 5144 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-lib-modules\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:39.998035 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:39.997864 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e292a072-7188-4a21-960f-ef5354c871c8-lib-modules\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:40.058878 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:40.058785 5144 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtd2l\" (UniqueName: \"kubernetes.io/projected/e292a072-7188-4a21-960f-ef5354c871c8-kube-api-access-qtd2l\") pod \"perf-node-gather-daemonset-6cmzh\" (UID: \"e292a072-7188-4a21-960f-ef5354c871c8\") " pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:40.124120 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:40.124077 5144 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:40.252074 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:40.252046 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785d4477d7-2j7w9_8843a38e-7855-4a1c-88c1-113c9c723ff5/manager/0.log" Nov 03 01:52:40.400513 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:40.400422 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8q225_2dc160d5-869c-4f83-9abc-0f3a0018740f/server/0.log" Nov 03 01:52:40.518009 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:40.517977 5144 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh"] Nov 03 01:52:40.519037 ip-10-0-143-162 kubenswrapper[5144]: W1103 01:52:40.519006 5144 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode292a072_7188_4a21_960f_ef5354c871c8.slice/crio-673dcf99146fecd2bcec32ed9f6e4b1c78ef0bcf04c5efe96f39ef6f5ba12b54 WatchSource:0}: Error finding container 673dcf99146fecd2bcec32ed9f6e4b1c78ef0bcf04c5efe96f39ef6f5ba12b54: Status 404 returned error can't find the container with id 673dcf99146fecd2bcec32ed9f6e4b1c78ef0bcf04c5efe96f39ef6f5ba12b54 Nov 03 01:52:41.269403 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:41.269357 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" event={"ID":"e292a072-7188-4a21-960f-ef5354c871c8","Type":"ContainerStarted","Data":"29ff106f5b73e1b85b2b0a6e52adeacb80e032427df69aecd47ad4d1a26fa365"} Nov 03 01:52:41.269977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:41.269428 5144 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" event={"ID":"e292a072-7188-4a21-960f-ef5354c871c8","Type":"ContainerStarted","Data":"673dcf99146fecd2bcec32ed9f6e4b1c78ef0bcf04c5efe96f39ef6f5ba12b54"} Nov 03 01:52:41.269977 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:41.269543 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:41.384436 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:41.384373 5144 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" podStartSLOduration=2.3843568 podStartE2EDuration="2.3843568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:52:41.383411444 +0000 UTC m=+6451.529105901" watchObservedRunningTime="2026-11-03 01:52:41.3843568 +0000 UTC m=+6451.530051255" Nov 03 01:52:41.527984 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:41.527889 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-m2z5c_916b609e-7a1f-4137-83b5-424605bc72b0/proxy/0.log" Nov 03 01:52:42.014423 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:42.014361 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-qx2hg_3ae333cb-5f6f-437f-9ba6-5cb249330692/manager/0.log" Nov 03 01:52:42.156370 ip-10-0-143-162 kubenswrapper[5144]: E1103 01:52:42.156335 5144 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-llwll" containerName="extractor" Nov 03 01:52:44.378525 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:44.378490 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-67cc775cd9-n49hj_af51a4fd-3ae1-46ac-b602-208c8a54c058/manager/0.log" Nov 03 01:52:46.123553 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:46.123508 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-785d4477d7-2j7w9_8843a38e-7855-4a1c-88c1-113c9c723ff5/manager/0.log" Nov 03 01:52:46.234439 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:46.234386 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-8q225_2dc160d5-869c-4f83-9abc-0f3a0018740f/server/0.log" Nov 03 01:52:47.287341 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:47.287313 5144 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26269/perf-node-gather-daemonset-6cmzh" Nov 03 01:52:47.630534 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:47.630445 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-m2z5c_916b609e-7a1f-4137-83b5-424605bc72b0/proxy/0.log" Nov 03 01:52:48.329236 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:48.329202 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-mbxcm_19f422bb-0c3f-4b3b-b8bd-72ef8d0e655d/cluster-monitoring-operator/0.log" Nov 03 01:52:50.775393 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:50.775361 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdp7m_29c9ab7f-0e92-400b-a88e-d4d53de42660/node-exporter/0.log" Nov 03 01:52:51.025598 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:51.025514 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj_e01a5e11-095c-402f-bfeb-07032cc4455c/istio-proxy/0.log" Nov 03 01:52:51.185122 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:51.185085 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdp7m_29c9ab7f-0e92-400b-a88e-d4d53de42660/kube-rbac-proxy/0.log" Nov 03 01:52:51.598815 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:51.598783 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bdp7m_29c9ab7f-0e92-400b-a88e-d4d53de42660/init-textfile/0.log" Nov 03 01:52:53.571217 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:53.571187 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54744779bc-6nckb_9b8e9601-f07e-4271-8d87-a6bfbbe91d9d/kube-auth-proxy/0.log" Nov 03 01:52:54.003828 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:54.003799 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-6jtgs_c4ebc0de-d5b0-4c2c-b4af-bdd212eb8c14/istio-proxy/0.log" Nov 03 01:52:54.745849 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:54.745814 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-fmkjq_9ea14538-faad-45d7-b7b8-8ac4e0362b0d/istio-proxy/0.log" Nov 03 01:52:55.385680 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:55.385647 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7bdc57c4cf-nfvvw_43f74203-a9e7-4cfe-ab2a-596a0ea37e1c/istio-proxy/0.log" Nov 03 01:52:56.237449 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:56.237418 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-c554b664c-lzst4_c7ce7db6-4c1e-4dce-a4cf-c735143ee8fa/istio-proxy/0.log" Nov 03 01:52:56.661468 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:56.661437 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-ccd6f9449-lp87k_5e2b6a92-c4bb-40a2-a630-ae33f705e854/payload-processing/0.log" Nov 03 01:52:56.884555 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:52:56.884519 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-767574d6c-vss4j_31e371fb-f718-43d0-bf0f-d909838ac2d0/router/0.log" Nov 03 01:53:01.964847 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:53:01.964804 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-64cfdbc956-4ltf2_a2768863-e2e0-46bd-9369-58c6a8a140d6/authorino/0.log" Nov 03 01:53:02.705328 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:53:02.705291 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-5f555b57d9lcdgj_e01a5e11-095c-402f-bfeb-07032cc4455c/istio-proxy/0.log" Nov 03 01:53:02.802156 ip-10-0-143-162 kubenswrapper[5144]: I1103 01:53:02.802124 5144 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-qx2hg_3ae333cb-5f6f-437f-9ba6-5cb249330692/manager/0.log"